Additive autoencoder for dimension estimation
Tommi Kärkkäinen, Jan Hänninen
Abstract
Dimension reduction is one of the key data transformation techniques in machine learning and knowledge discovery. It can be realized by using linear and nonlinear transformation techniques. An additive autoencoder for dimension reduction, which is composed of a serially performed bias estimation, linear trend estimation, and nonlinear residual estimation, is proposed and analyzed. Compared to the classical model, adding an explicit linear operator to the overall transformation and considering the nonlinear residual estimation in the original data dimension significantly improves the data reproduction capabilities of the proposed model. The computational experiments confirm that an autoencoder of this form, with only a shallow network to encapsulate the nonlinear behavior, is able to identify an intrinsic dimension of a dataset with low autoencoding error. This observation leads to an investigation in which shallow and deep network structures, and how they are trained, are compared. We conclude that the deeper network structures obtain lower autoencoding errors during the identification of the intrinsic dimension. However, the detected dimension does not change compared to a shallow network. As far as we know, this is the first experimental result concluding no benefit from a deep architecture compared to its shallow counterpart.