Imputation of Missing PM2.5 Observations in a Network of Air Quality Monitoring Stations by a New kNN Method
Idit Belachsen, David M. Broday
Abstract
Statistical analyses often require unbiased and reliable data completion. In this work, we imputed missing fine particulate matter (PM2.5) observations from eight years (2012–2019) of records in 59 air quality monitoring (AQM) stations in Israel, using no auxiliary data but the available PM2.5 observations. This was achieved by a new k-Nearest Neighbors multivariate imputation method (wkNNr) that uses the correlations between the AQM stations’ data to weigh the distance between the observations. The model was evaluated against an iterative imputation with an Ensemble of Extremely randomized decision Trees (iiET) on artificially and randomly removed data intervals of various lengths: very short (0.5–3 h, corresponding to 1–6 missing values), short (6–24 h), medium-length (36–72 h), long (10–30 d), and very long (30 d–2 y). The new wkNNr model outperformed the iiET in imputing very short missing-data intervals when the adjacent lagging and leading observations were added as model inputs. For longer missing-data intervals, despite its simplicity and the smaller number of hyperparameters required for tuning, the new model showed an almost comparable performance to the iiET. A parallel Python implementation of the new kNN-based multivariate imputation method is available on github.