In statistics, missing data, or missing values, occur when no data value is stored for the variable in an observation. Missing data are a common occurrence and can have a significant effect on the conclusions that can be drawn from the data.
Missing data can occur because of nonresponse: no information is provided for one or more items or for a whole unit ("subject"). Some items are more likely to generate a nonresponse than others: for example items about private subjects such as income. Attrition is a type of missingness that can occur in longitudinal studies—for instance studying development where a measurement is repeated after a certain period of time. Missingness occurs when participants drop out before the test ends and one or more measurements are missing.
Data often are missing in research in economics, sociology, and political science because governments or private entities choose not to, or fail to, report critical statistics,[1] or because the information is not available. Sometimes missing values are caused by the researcher—for example, when data collection is done improperly or mistakes are made in data entry.[2]
These forms of missingness take different types, with different impacts on the validity of conclusions from research: Missing completely at random, missing at random, and missing not at random. Missing data can be handled similarly as censored data.
Types
Understanding the reasons why data are missing is important for handling the remaining data correctly. If values are missing completely at random, the data sample is likely still representative of the population. But if the values are missing systematically, analysis may be biased. For example, in a study of the relation between IQ and income, if participants with an above-average IQ tend to skip the question ‘What is your salary?’, analyses that do not take into account this missing at random (MAR pattern (see below)) may falsely fail to find a positive association between IQ and salary. Because of these problems, methodologists routinely advise researchers to design studies to minimize the occurrence of missing values.[2] Graphical models can be used to describe the missing data mechanism in detail.[3][4]
The graph shows the probability distributions of the estimations of the expected intensity of depression in the population. The number of cases is 60. Let the true population be a standardised normal distribution and the non-response probability be a logistic function of the intensity of depression. The conclusion is: The more data is missing (MNAR), the more biased are the estimations. We underestimate the intensity of depression in the population.
Missing completely at random
Values in a data set are missing completely at random (MCAR) if the events that lead to any particular data-item being missing are independent both of observable variables and of unobservable parameters of interest, and occur entirely at random.[5] When data are MCAR, the analysis performed on the data is unbiased; however, data are rarely MCAR.
In the case of MCAR, the missingness of data is unrelated to any study variable: thus, the participants with completely observed data are in effect a random sample of all the participants assigned a particular intervention. With MCAR, the random assignment of treatments is assumed to be preserved, but that is usually an unrealistically strong assumption in practice.[6]
Missing at random
Missing at random (MAR) occurs when the missingness is not random, but where missingness can be fully accounted for by variables where there is complete information.[7] Since MAR is an assumption that is impossible to verify statistically, we must rely on its substantive reasonableness.[8] An example is that males are less likely to fill in a depression survey but this has nothing to do with their level of depression, after accounting for maleness. Depending on the analysis method, these data can still induce parameter bias in analyses due to the contingent emptiness of cells (male, very high depression may have zero entries). However, if the parameter is estimated with Full Information Maximum Likelihood, MAR will provide asymptotically unbiased estimates.[citation needed]
Missing not at random
Missing not at random (MNAR) (also known as nonignorable nonresponse) is data that is neither MAR nor MCAR (i.e. the value of the variable that's missing is related to the reason it's missing).[5] To extend the previous example, this would occur if men failed to fill in a depression survey because of their level of depression.
Techniques of dealing with missing data
Missing data reduces the representativeness of the sample and can therefore distort inferences about the population. Generally speaking, there are three main approaches to handle missing data: (1) Imputation—where values are filled in the place of missing data, (2) omission—where samples with invalid data are discarded from further analysis and (3) analysis—by directly applying methods unaffected by the missing values. One systematic review addressing the prevention and handling of missing data for patient-centered outcomes research identified 10 standards as necessary for the prevention and handling of missing data. These include standards for study design, study conduct, analysis, and reporting.[9]
In some practical application, the experimenters can control the level of missingness, and prevent missing values before gathering the data. For example, in computer questionnaires, it is often not possible to skip a question. A question has to be answered, otherwise one cannot continue to the next. So missing values due to the participant are eliminated by this type of questionnaire, though this method may not be permitted by an ethics board overseeing the research. In survey research, it is common to make multiple efforts to contact each individual in the sample, often sending letters to attempt to persuade those who have decided not to participate to change their minds.[10]:161–187 However, such techniques can either help or hurt in terms of reducing the negative inferential effects of missing data, because the kind of people who are willing to be persuaded to participate after initially refusing or not being home are likely to be significantly different from the kinds of people who will still refuse or remain unreachable after additional effort.[10]:188–198
In situations where missing values are likely to occur, the researcher is often advised on planning to use methods of data analysis methods that are robust to missingness. An analysis is robust when we are confident that mild to moderate violations of the technique's key assumptions will produce little or no bias, or distortion in the conclusions drawn about the population.
Imputation
Main article: Imputation (statistics)
Some data analysis techniques are not robust to missingness, and require to "fill in", or impute the missing data. Rubin (1987) argued that repeating imputation even a few times (5 or less) enormously improves the quality of estimation.[2] For many practical purposes, 2 or 3 imputations capture most of the relative efficiency that could be captured with a larger number of imputations. However, a too-small number of imputations can lead to a substantial loss of statistical power, and some scholars now recommend 20 to 100 or more.[11] Any multiply-imputed data analysis must be repeated for each of the imputed data sets and, in some cases, the relevant statistics must be combined in a relatively complicated way.[2]
The expectation-maximization algorithm is an approach in which values of the statistics which would be computed if a complete dataset were available are estimated (imputed), taking into account the pattern of missing data. In this approach, values for individual missing data-items are not usually imputed.
Interpolation(example:bilinear interpolation)
Main article: Interpolation
In the mathematical field of numerical analysis, interpolation is a method of constructing new data points within the range of a discrete set of known data points.
In the comparison of two paired samples with missing data, a test statistic that uses all available data without the need for imputation is the partially overlapping samples t-test.[12] This is valid under normality and assuming MCAR
Partial deletion
Methods which involve reducing the data available to a dataset having no missing values include:
Listwise deletion/casewise deletion
Pairwise deletion
Full analysis
Methods which take full account of all information available, without the distortion resulting from using imputed values as if they were actually observed:
Generative approaches:
The expectation-maximization algorithm
full information maximum likelihood estimation
Discriminative approaches:
Max-margin classification of data with absent features [13][14]
Partial identification methods may also be used.[15]
Model-based techniques
Model based techniques, often using graphs, offer additional tools for testing missing data types (MCAR, MAR, MNAR) and for estimating parameters under missing data conditions. For example, a test for refuting MAR/MCAR reads as follows:
For any three variables X,Y, and Z where Z is fully observed and X and Y partially observed, the data should satisfy: \( X\perp \!\!\!\perp R_{y}|(R_{x},Z). \)
In words, the observed portion of X should be independent on the missingness status of Y, conditional on every value of Z. Failure to satisfy this condition indicates that the problem belongs to the MNAR category.[16]
(Remark: These tests are necessary for variable-based MAR which is a slight variation of event-based MAR.[17][18][19])
When data falls into MNAR category techniques are available for consistently estimating parameters when certain conditions hold in the model.[3] For example, if Y explains the reason for missingness in X and Y itself has missing values, the joint probability distribution of X and Y can still be estimated if the missingness of Y is random. The estimand in this case will be:
\( {\displaystyle {\begin{aligned}P(X,Y)&=P(X|Y)P(Y)\\&=P(X|Y,R_{x}=0,R_{y}=0)P(Y|R_{y}=0)\end{aligned}}} \)
where \( R_{x}=0 \)and \( R_{y}=0 \) denote the observed portions of their respective variables.
Different model structures may yield different estimands and different procedures of estimation whenever consistent estimation is possible. The preceding estimand calls for first estimating P ( X | Y ) {\displaystyle P(X|Y)} P(X|Y) from complete data and multiplying it by P ( Y ) {\displaystyle P(Y)} P(Y) estimated from cases in which Y is observed regardless of the status of X. Moreover, in order to obtain a consistent estimate it is crucial that the first term be P ( X | Y ) {\displaystyle P(X|Y)} P(X|Y) as opposed to P ( Y | X ) {\displaystyle P(Y|X)} P(Y|X).
In many cases model based techniques permit the model structure to undergo refutation tests.[19] Any model which implies the independence between a partially observed variable X and the missingness indicator of another variable Y (i.e. \( R_{y}) \), conditional on\( R_{x} \) can be submitted to the following refutation test: \( X\perp \!\!\!\perp R_{y}|R_{x}=0. \)
Finally, the estimands that emerge from these techniques are derived in closed form and do not require iterative procedures such as Expectation Maximization that are susceptible to local optima.[20]
A special class of problems appears when the probability of the missingness depends on time. For example, in the trauma databases the probability to lose data about the trauma outcome depends on the day after trauma. In these cases various non-stationary Markov chain models are applied. [21]
See also
Censoring (statistics)
Expectation–maximization algorithm
Indicator variable
Inverse probability weighting
Latent variable
Matrix completion
References
Messner SF (1992). "Exploring the Consequences of Erratic Data Reporting for Cross-National Research on Homicide". Journal of Quantitative Criminology. 8 (2): 155–173. doi:10.1007/bf01066742. S2CID 133325281.
Hand, David J.; Adèr, Herman J.; Mellenbergh, Gideon J. (2008). Advising on Research Methods: A Consultant's Companion. Huizen, Netherlands: Johannes van Kessel. pp. 305–332. ISBN 978-90-79418-01-5.
Mohan, Karthika; Pearl, Judea; Tian, Jin (2013). Advances in Neural Information Processing Systems 26. pp. 1277–1285.
Karvanen, Juha (2015). "Study design in causal models". Scandinavian Journal of Statistics. 42 (2): 361–377. arXiv:1211.2958. doi:10.1111/sjos.12110. S2CID 53642701.
Polit DF Beck CT (2012). Nursing Research: Generating and Assessing Evidence for Nursing Practice, 9th ed. Philadelphia, USA: Wolters Klower Health, Lippincott Williams & Wilkins.
Deng (2012-10-05). "On Biostatistics and Clinical Trials". Archived from the original on 15 March 2016. Retrieved 13 May 2016.
"Archived copy". Archived from the original on 2015-09-10. Retrieved 2015-08-01.
Little, Roderick J. A.; Rubin, Donald B. (2002), Statistical Analysis with Missing Data (2nd ed.), Wiley.
Li, Tianjing; Hutfless, Susan; Scharfstein, Daniel O.; Daniels, Michael J.; Hogan, Joseph W.; Little, Roderick J.A.; Roy, Jason A.; Law, Andrew H.; Dickersin, Kay (2014). "Standards should be applied in the prevention and handling of missing data for patient-centered outcomes research: a systematic review and expert consensus". Journal of Clinical Epidemiology. 67 (1): 15–32. doi:10.1016/j.jclinepi.2013.08.013. PMC 4631258. PMID 24262770.
Stoop, I.; Billiet, J.; Koch, A.; Fitzgerald, R. (2010). Reducing Survey Nonresponse: Lessons Learned from the European Social Survey. Oxford: Wiley-Blackwell. ISBN 978-0-470-51669-0.
Graham J.W.; Olchowski A.E.; Gilreath T.D. (2007). "How Many Imputations Are Really Needed? Some Practical Clarifications of Multiple Imputation Theory". Preventative Science. 8 (3): 208–213. CiteSeerX 10.1.1.595.7125. doi:10.1007/s11121-007-0070-9. PMID 17549635. S2CID 24566076.
Derrick, B; Russ, B; Toher, D; White, P (2017). "Test Statistics for the Comparison of Means for Two Samples That Include Both Paired and Independent Observations". Journal of Modern Applied Statistical Methods. 16 (1): 137–157. doi:10.22237/jmasm/1493597280.
Chechik, Gal; Heitz, Geremy; Elidan, Gal; Abbeel, Pieter; Koller, Daphne (2008-06-01). "Max-margin Classification of incomplete data" (PDF). Neural Information Processing Systems: 233–240.
Chechik, Gal; Heitz, Geremy; Elidan, Gal; Abbeel, Pieter; Koller, Daphne (2008-06-01). "Max-margin Classification of Data with Absent Features". The Journal of Machine Learning Research. 9: 1–21. ISSN 1532-4435.
Tamer, Elie (2010). "Partial Identification in Econometrics". Annual Review of Economics. 2 (1): 167–195. doi:10.1146/annurev.economics.050708.143401.
Mohan, Karthika; Pearl, Judea (2014). "On the testability of models with missing data". Proceedings of AISTAT-2014, Forthcoming.
Darwiche, Adnan (2009). Modeling and Reasoning with Bayesian Networks. Cambridge University Press.
Potthoff, R.F.; Tudor, G.E.; Pieper, K.S.; Hasselblad, V. (2006). "Can one assess whether missing data are missing at random in medical studies?". Statistical Methods in Medical Research. 15 (3): 213–234. doi:10.1191/0962280206sm448oa. PMID 16768297. S2CID 12882831.
Pearl, Judea; Mohan, Karthika (2013). Recoverability and Testability of Missing data: Introduction and Summary of Results (PDF) (Technical report). UCLA Computer Science Department, R-417.
Mohan, K.; Van den Broeck, G.; Choi, A.; Pearl, J. (2014). "An Efficient Method for Bayesian Network Parameter Learning from Incomplete Data". Presented at Causal Modeling and Machine Learning Workshop, ICML-2014.
Mirkes, E.M.; Coats, T.J.; Levesley, J.; Gorban, A.N. (2016). "Handling missing data in large healthcare dataset: A case study of unknown trauma outcomes". Computers in Biology and Medicine. 75: 203–216. arXiv:1604.00627. Bibcode:2016arXiv160400627M. doi:10.1016/j.compbiomed.2016.06.004. PMID 27318570. S2CID 5874067. Archived from the original on 2016-08-05.
Further reading
Acock AC (2005), "Working with missing values", Journal of Marriage and Family, 67 (4): 1012–28, doi:10.1111/j.1741-3737.2005.00191.x, archived from the original on 2013-01-05
Allison, Paul D. (2001), Missing Data, SAGE Publishing
Bouza-Herrera, Carlos N. (2013), Handling Missing Data in Ranked Set Sampling, Springer
Enders, Craig K. (2010), Applied Missing Data Analysis, Guilford Press
Graham, John W. (2012), Missing Data, Springer
Molenberghs, Geert; Fitzmaurice, Garrett; Kenward, Michael G.; Tsiatis, Anastasios; Verbeke, Geert, eds. (2015), Handbook of Missing Data Methodology, Chapman & Hall
Raghunathan, Trivellore (2016), Missing Data Analysis in Practice, Chapman & Hall
Little, Roderick J. A.; Rubin, Donald B. (2002), Statistical Analysis with Missing Data (2nd ed.), Wiley
Tsiatis, Anastasios A. (2006), Semiparametric Theory and Missing Data, Springer
Van den Broeck J, Cunningham SA, Eeckels R, Herbst K (2005), "Data cleaning: detecting, diagnosing, and editing data abnormalities", PLOS Medicine, 2 (10): e267, doi:10.1371/journal.pmed.0020267, PMC 1198040, PMID 16138788, S2CID 5667073
Zarate LE, Nogueira BM, Santos TR, Song MA (2006). "Techniques for Missing Value Recovering in Imbalanced Databases: Application in a marketing database with massive missing data". IEEE International Conference on Systems, Man and Cybernetics, 2006. SMC '06. 3. pp. 2658–2664. doi:10.1109/ICSMC.2006.385265.
Statistics
Outline Index
Descriptive statistics
Continuous data
Center
Mean
arithmetic geometric harmonic Median Mode
Dispersion
Variance Standard deviation Coefficient of variation Percentile Range Interquartile range
Shape
Central limit theorem Moments
Skewness Kurtosis L-moments
Count data
Index of dispersion
Summary tables
Grouped data Frequency distribution Contingency table
Dependence
Pearson product-moment correlation Rank correlation
Spearman's ρ Kendall's τ Partial correlation Scatter plot
Graphics
Bar chart Biplot Box plot Control chart Correlogram Fan chart Forest plot Histogram Pie chart Q–Q plot Run chart Scatter plot Stem-and-leaf display Radar chart Violin plot
Data collection
Study design
Population Statistic Effect size Statistical power Optimal design Sample size determination Replication Missing data
Survey methodology
Sampling
stratified cluster Standard error Opinion poll Questionnaire
Controlled experiments
Scientific control Randomized experiment Randomized controlled trial Random assignment Blocking Interaction Factorial experiment
Adaptive Designs
Adaptive clinical trial Up-and-Down Designs Stochastic approximation
Observational Studies
Cross-sectional study Cohort study Natural experiment Quasi-experiment
Statistical inference
Statistical theory
Population Statistic Probability distribution Sampling distribution
Order statistic Empirical distribution
Density estimation Statistical model
Model specification Lp space Parameter
location scale shape Parametric family
Likelihood (monotone) Location–scale family Exponential family Completeness Sufficiency Statistical functional
Bootstrap U V Optimal decision
loss function Efficiency Statistical distance
divergence Asymptotics Robustness
Frequentist inference
Point estimation
Estimating equations
Maximum likelihood Method of moments M-estimator Minimum distance Unbiased estimators
Mean-unbiased minimum-variance
Rao–Blackwellization Lehmann–Scheffé theorem Median unbiased Plug-in
Interval estimation
Confidence interval Pivot Likelihood interval Prediction interval Tolerance interval Resampling
Bootstrap Jackknife
Testing hypotheses
1- & 2-tails Power
Uniformly most powerful test Permutation test
Randomization test Multiple comparisons
Parametric tests
Likelihood-ratio Score/Lagrange multiplier Wald
Specific tests
Z-test (normal) Student's t-test F-test
Goodness of fit
Chi-squared G-test Kolmogorov–Smirnov Anderson–Darling Lilliefors Jarque–Bera Normality (Shapiro–Wilk) Likelihood-ratio test Model selection
Cross validation AIC BIC
Rank statistics
Sign
Sample median Signed rank (Wilcoxon)
Hodges–Lehmann estimator Rank sum (Mann–Whitney) Nonparametric anova
1-way (Kruskal–Wallis) 2-way (Friedman) Ordered alternative (Jonckheere–Terpstra)
Bayesian inference
Bayesian probability
prior posterior Credible interval Bayes factor Bayesian estimator
Maximum posterior estimator
CorrelationRegression analysis
Correlation
Pearson product-moment Partial correlation Confounding variable Coefficient of determination
Regression analysis
Errors and residuals Regression validation Mixed effects models Simultaneous equations models Multivariate adaptive regression splines (MARS)
Linear regression
Simple linear regression Ordinary least squares General linear model Bayesian regression
Non-standard predictors
Nonlinear regression Nonparametric Semiparametric Isotonic Robust Heteroscedasticity Homoscedasticity
Generalized linear model
Exponential families Logistic (Bernoulli) / Binomial / Poisson regressions
Partition of variance
Analysis of variance (ANOVA, anova) Analysis of covariance Multivariate ANOVA Degrees of freedom
Categorical / Multivariate / Time-series / Survival analysis
Categorical
Cohen's kappa Contingency table Graphical model Log-linear model McNemar's test
Multivariate
Regression Manova Principal components Canonical correlation Discriminant analysis Cluster analysis Classification Structural equation model
Factor analysis Multivariate distributions
Elliptical distributions
Normal
Time-series
General
Decomposition Trend Stationarity Seasonal adjustment Exponential smoothing Cointegration Structural break Granger causality
Specific tests
Dickey–Fuller Johansen Q-statistic (Ljung–Box) Durbin–Watson Breusch–Godfrey
Time domain
Autocorrelation (ACF)
partial (PACF) Cross-correlation (XCF) ARMA model ARIMA model (Box–Jenkins) Autoregressive conditional heteroskedasticity (ARCH) Vector autoregression (VAR)
Frequency domain
Spectral density estimation Fourier analysis Wavelet Whittle likelihood
Survival
Survival function
Kaplan–Meier estimator (product limit) Proportional hazards models Accelerated failure time (AFT) model First hitting time
Hazard function
Nelson–Aalen estimator
Test
Log-rank test
Applications
Biostatistics
Bioinformatics Clinical trials / studies Epidemiology Medical statistics
Engineering statistics
Chemometrics Methods engineering Probabilistic design Process / quality control Reliability System identification
Social statistics
Actuarial science Census Crime statistics Demography Econometrics Jurimetrics National accounts Official statistics Population statistics Psychometrics
Spatial statistics
Cartography Environmental statistics Geographic information system Geostatistics Kriging
Undergraduate Texts in Mathematics
Graduate Studies in Mathematics
Hellenica World - Scientific Library
Retrieved from "http://en.wikipedia.org/"
All text is available under the terms of the GNU Free Documentation License
