Guidelines for Data Quality and Preprocessing
- 1. Week 2 316 – Data quality Dr Patrick Mukala
- 2. 2 Getting and Cleaning Data overview Getting Cleaning Storing Hadoop/Hive (SQL) MongoDB (NoSQL) Data Quality Missing Data MapReduce
- 3. 3 Data Quality Data Quality measures: • Validity • Accuracy • Completeness • Consistency • Uniformity
- 4. 4 Missing Data origins (Types) • Not applicable, N/A • Not available (i.e. variable added to questionnaire at a later date, missing data due to defect) • Unknown • Refusal to answer • True Missing (i.e. question skipped) Analyze missing data to determine how to deal with missing data.
- 5. 5 Missing Data mechanisms 1. Missing completely at random (MCAR): missing data on Y is unrelated to the value of Y itself or to the values of any other variable in the data set 2. Missing at random (MAR): The probability of missing data on Y is unrelated to the value of Y after controlling for other variables in the analysis (say X). 3. Not missing at random (NMAR): Missing values do depend on unobserved values.
- 6. 6 Example MCAR • We want to assess which are the main determinants of income (such as age). The MCAR assumption would be violated if people who did not report their income were, on average, younger than people who reported it. This can be tested by dividing the sample into those who did and did not report their income, and then testing a difference in mean age. • Water damage of paper test results prior to entry
- 7. 7 Example MAR • The MAR assumption would be satisfied if the probability of missing data on income depended on a person’s age, but within age group the probability of missing income was unrelated to income. However, this cannot be tested because we do not know the values of the missing data, thus, we cannot compare the values of those with and without missing data to see if they systematically differ on that variable. • For really sick patients, clinicians may not draw blood for lab analysis. • Titanic: NaN’s for age entry for travellers of class 3
- 8. 8 Example NMAR • The NMAR assumption would be fulfilled if people with high income are less likely to report their income. • Titanic: NaN’s for cabin entry for travellers of class 3
- 9. 9 Options for handling Missing Data • Listwise deletion (or complete case analysis) • Imputation methods • Marginal mean imputation • Conditional mean imputation
- 10. 10 Handling Missing Data • First, do no harm. Use best practices and careful methodology to minimize missingness • Be transparent. Report any incidences of missing data • Explicitly discuss whether data are missing at random • Discuss how you as a researcher have dealt with the issue of incomplete data and the results of your intervention
Editor's Notes
- #5: Missing completely at random (MCAR): data are missing independently of both observed and unobserved data. Example: a participant flips a coin to decide whether to complete the depression survey. Missing at random (MAR): given the observed data, data are missing independently of unobserved data. Example: male participants are more likely to refuse to fill out the depression survey, but it does not depend on the level of their depression. MCAR implies MAR, but not the other way round. Most methods assume MAR. We can ignore missing data ( = omit missing observations) if we have MAR or MCAR. Missing Not at Random (MNAR): missing observations related to values of unobserved data. Example: participants with severe depression, or side-effects from the medication, were more likely to be missing at end.