Notes

Data Exploration and Data Preparation The phases of exploration, cleaning and preparation of the headset can cover the 70% of the total time of a data mining project. The steps are summarized as follows:

• Identification Of Variables
  • First we need to identify the variables Predictors (Inputs) and Target (Output).
  • Secondly, we must recognize the type and level of data measurement, example:

• The variables in the previous slide table may be reclassified as follows:

• Univariate Analysis
  • In this phase the variables are analyzed one at a time.
  • The type of analysis depends on the nature of the variable> categorical or numerical/continuous.
  • For numerical/continuous variables, you must first understand the values of central tendency and variability:

• Univariate analysis is also useful to highlight missing and outlier data, which will be discussed specifically later.

• For categorical variables, frequency tables will be used to understand the distribution of each category (values and absolute or percentage frequencies); the bar chart can be used as a graphic representation.

• Bivariate Analysis

  • The bivariate analysis shows the relationship between two variables.
  • In this case we try to evaluate whether there is an association or independence between the variables, considering a specific significance level.
  • We can perform bivariate analysis for any combination of categorical and numerical/continuous variables.
  • The combinations can be:
    • Categorical vs categorical
      • In this case we use a contingency table and stacked bar charts.
      • With contingency tables, we analyze the absolute or percentage joint frequencies of the observed values of the two variables; on the rows we will find the categories of a variable and on the columns the categories of the second variable.
      • The stacked column chart enables a visual representation of the relationship.
      • The significance test used is the chi-square, and to examine the strength of the relationship using indices such as the V Cramer.
    • Categorical vs numerical/continuous
      • In this case, we typically use box plots of the numeric variable for each level of the categorical variable.
      • To verify the significance of the relation, we can use the t-test, if the number of the qualitative categories is equal to 2, or F-Test if the number of categories is greater than 2.
    • Continuous vs continuous
      • In this case you use a scatter diagram and the linear correlation coefficient, to find the strength of the relationship.
      • The correlation varies between -1 and +1.
        • -1: Perfect negative linear correlation.
        • +1: Perfect positive linear correlation.
        • 0: No correlation.
  • Several methods are used to analyze these combinations during the data preparation process.

• Missing Data Management

  • The presence of missing data in the training dataset may reduce the fit and predictive ability of the model, possibly leading to incorrect forecasts and mis-classification.
  • Missing values consequences on predictive models:
    • Considering the precedent tables, in the left-wing scenario, we did not treat missing values.
    • The inference from this dataset is that the ability of males to play cricket is equivalent to that of females.
    • If you look at the second table, showing data after treatment of missing values (based on gender), we can see that females are more likely to play cricket than males.
  • Missing values Sources:
    • The origin of the missing data can be essentially linked to two stages of the process:
      • Data Extraction
      • Data Collection
    • Considering the first stage mentioned, we should conduct a double check in the extraction stage, possibly using the "data type guardians software" and/or using transformation of the extracted strings procedures (hashing procedures) that help to eliminate or minimize such extraction errors.
    • As for the second one, the problems are certainly more complex and they can have various configurations.
  • Missing values Configurations:
    • Completely Random Missing, in this case, the probability of having a missing data is identical for each observation.
    • Random Missing, in this case, the presence of missing data in a variable is random, but has a relationship with other input variables (ex. The missing data in the age variable is more common for women than men).
    • Missing that depend on Predictors Not Observed, in this case, the missing data are not random, but depend on a variable that has not been inserted in the database. Ex. If in a clinical study a diagnostic procedure may cause discomfort in some patients, they may leave the study prematurely, generating a non-random missing profile, unless we iontroduce the "uncomfortable" missing variable to control themselves.
    • Missing that depend on the Same Value of the Variable, in this case, the probability of missing depends on the missing value in itself. Ex. Who has a low income tend to not declare their income in a survey.
  • Missing value How To Process:
    • Deletion of Missing Data, there are two ways of deleting missing data: Listwise or Pairwise
      • In Listwise deletion, the entire observation that contains the missing data is deleted. Clearly it is the easiest approach to the database clean, but it has the disadvantage of markedly reduce the sample size.
      • In the Pairwise deletion, any statistical analysis is calculated with all the valid cases present in each variable, eliminating the observation only if one of the variables involved containts a giving missing. The advantage is to preserve degrees of freedom, the disadvantage is that the sample size may be different for different variables involved in the analysis.

This type of deletion is used when missing data are totally random, otherwise we could invalidate the predictive ability of the model.

• Assignment with mean / mode / median: this approach replaces the missing data with estimated values. The objective is to use the relationships between variables in the dataset to estimate as accurately as possible the missing data itself. The mean and median are used for numeric data, the mode for categorical data:
  • Generalized replacement, in this case we calculate the mean or median for all valid data available and replace all missing data with these values. For example in the table above the average of valid cases for Manpower is 28,33 and then the missing data is replaced with that value.
  • Similarity replacement, in this case, we calculate separately the average of Labour for men (29.75) and women (25) and replace missing values taking into account the genre to which the observation with missing data relate.
• Building Predictive Models, the setup of a forecasting model is one of the most sophisticated method for handling missing data: it creates a predictive model to estimate the values that will replace the missing data.

In this case, the interested data are divided into two groups: a set without missing values for the investigated variable and another with missing values: the first dataset becomes the set of the model training data, while the second set with values missing is set as test data, and therefore the variable with missing values are treated as target.

• Predictive Models, a model is then estimated to predict the target variable based on other attributes of the training dataset and to populate the missing values of the target variable; the techniques used are regression, ANOVA, logistic regression and various other modeling techniques.

There are two disadvantages in this approach:

• The model estimated values are usually more regular than the true values.
• If there are no relationships between the variables in the dataset and the data with missing values, the model will be inaccurate in estimating missing values themselves.
• Recording with KNN Neural Network, with this allocation method, the missing attribute values are calculated using the similarity pattern between the observations of several variables, on the basis of an appropriate distance function between those patterns; the missing value of a variable is therefore replaced with the "closer" valid value. The advantages of the method:
  • It can be used interchangeably to qualitative and quantitative variables.
  • It is not necessary to build a predictive model for each attribute.
  • We can treat cases with missing values of a different nature.
  • It is considered the structure of correlation between the variables.

The disadvantages are related to the computational difficulty of large datasets and the sensitivity of the model parametrization.

• Outlier Treatment
  • Outliers Definition
    • Outlier detection is commonly used by analysts and data scientists; the problem of outliers needs a lot of attention, otherwise you can get the production of heavily wrong estimates in predictive models.
    • An outlier is an abnormal value, an observation that appears distant and deviates from a typical pattern of a given sample.
    • Example: customer profiling, it turns out that the average annual income of customers is $0.8 million, but two customers have annual incomes of $4 and $4.2 million. These two clients have therefore an annual income much higher than the rest of the population and will be seen as outliers.
  • Outliers Types
    • Outliers can be of two types: univariate and multi variate. In the slide before, we discussed the example of univariate outliers. These outliers can be found by observing the distribution of a single variable, and multivariate outliers are outliers in a n-dimensional space. In order to detect them, the multi-dimensional distribution must be examined.
    • Example, here we can observe the univariate and bivariate distribution for height, weight. Looking at the box plots, we have no abnormal value (above and below 1.5 * IQR, the most common method). Instead, looking at the scatter plot we notice two lower and one higher than the average values in a specific segment of the weight and height.
  • Outliers Origins
    • Data Entry Errors, human errors, such as errors caused during data collection, recording, or inputs can generate atypical values in the same data.
    • Measurement Error, is the most common source of outliers. It occurs when the measurement tool used is faulty. Example, 10 weighing machines. 9 of them are correct, one is faulty. The weight measured on a faulty machine will be higher / lower than the rest of the measurements in the gourds. The weights measured on faulty machines can lead to abnormal values.
    • Experimental Error, another cause of outliers is the experimental error, caused by an abnormal event that has affected the outcome of the experiment itself.
    • Intentional Outlier, in general are related to sensitive data. For example: imagine you interview some young people on alcohol consumption. Only some of them will report the actual value; in this case the actual values may appear as outliers.
    • Data Processing Error, usually in data mining activities, we extract data from more sources. It is possible that some errors of manipulation or extraction bring outliers in the set of final data.
    • Sampling Error, for example, we have to measure the height of few athletes. For error, we include a pair of basket players in the sample. This inclusion can cause abnormal values in the set of data.
    • Natural Outlier, when an outlier is not artificial (caused by one of the previously identified errors), is precisely a natural erratic value.
  • Outlier Consequences
    • Increase the error variance and reduce the power of the statistical tests.
    • If outliers are not distributed randomly, they can compromise the normality of some distributions.
    • They can heavily influence the estimates of interest.
    • They may have an impact on the basic assumptions of regression, ANOVA and other statistical models.
    • Example:

As you can see, the data with outliers have significantly different mean and standard deviation. In the first scenario, the average is 5.45. But with the outlier, the average increases to 30. This changes completely our estimate of the central distribution location.

• Outliers, How To Identify Them, the most commonly used methods to detect outliers are charts, such as Box Plot, Histogram, Scatter Plot (previously we used box plots and scatter plots to display).

Some analysts also various rules of thumb to detect outliers:

• Values that are beyond the limits of -1.5 x 1.5 x IQR.
• Values outside the range between the 5^th^ and 95^th^ percentile.
• Values far three or more standard deviations from the mean.
• Identification of influential points, through appropriate influence or leverage ratios, such as Mahalanobis distance and Cook's D.
• Outliers, How To Remove, most of the ways to put a remedy to the anomalies of the data are similar to the methods used to treat the missing values:
  • Elimination of Observations, it is appropriate if they are the result of a data entry error or if the outliers are relatively few in number, we can also cut the extreme tails of the distribution.
  • Data Transformations, logarithmic transformation, grouping into categories, reducing their impact on the analysis (weighting).
  • Treating them as a Separate Group, in the presence of a significant number of abnormal data, it may be appropriate to separate them from the rest of the observations, estimate an ad hoc predictive model and integrate it with the base model.
  • Replacing values, as in the case of the missing data, we can replace with mean, mode or specific predictive models.

Variables Transformation

• Motivation and Methods

  In modeling, the variable transformation involves the replacement of the variable with a function of the variable itself: for example, the square, the square root, the cube root or the logarithm x is a transformation. Motivations to proceed into a variable transformation:

• When you want to change the scale of a variable or standardize his values. Sometimes this transformation is essential if you are working with data expressed in different scales, in any case do not change the shape of the distribution of the variable.
• When we can not transform complex linear relationships in linear relationships. The existence of a linear relationship between variables is easier to estimate and manage than a non-linear. The scatter plot can be used to find the relationship between two continuous variables; these transformations also improve the prediction. The logarithmic transformation is one of the commonly used processing techniques for this purpose.

• When a symmetrical distribution is preferred to an asymmetrical: some predictive models require the normality of the variables used and the transformations of variables may at least partially remove the asymmetry; in particular for right asymmetric distribution you can use the square root transformations or cubic or the logarithm, for those left oblique the square, the cube or the exponential.
• When business evaluations lead us to conclude that is significant for the analysis and for the deployment of the model to use appropriate categorizations of the variables (ex. Age group, household income classes, etc.)
• When you want to use categorical variables in models that accept only numerical variables (transformation of categorical variables into dummy variables -- 0/1).
• Creation of New Variables