Enhancing Expectation-Maximization for Missing Data Imputation: Machine Learning and Probabilistic Approaches
Sangeetha Mani, Pravin R. Kshirsagar, Tan Kuan Tak, Shrikant V. Sonekar
Abstract
In the era of Artificial Intelligence and Machine Learning, data analysis is attracting higher attention in different areas which include finance, healthcare, weather systems and IOT applications. As all those areas are working with datasets and looking after the insights of datasets, it is mandatory that the dataset should be clear and error free as it may impact the decisions wrongly if there are any outliers. To treat the dataset which have the missing data, different algorithms has evolved to keep up the dataset accurate after imputing the missing data. However, many algorithms fail to do so and Expecation-Maximization (EM) algorithm proved to produce accurate dataset after imputation by preserving its structural integrity. The EM can work with complex and high dimensional dataset; however, the accuracy of the imputed data is reduced when the EM algorithm is applied with high dimensional datasets. Hence, an adaptive method termed as Enhanced Expectation-maximization algorithm by introducing adaptive learning rate is proposed in this paper. The adaptive learning rate will be dynamically updated on each iteration. The algorithm provided higher accuracy with lesser Root Mean Square Error (RMSE).