LASSO-based feature selection for improved microbial and microbiome classification
Owen Queen, Scott Emrich
Abstract
Microbial and microbiome data consist of large data matrices that contain many irrelevant or colinear features. This can hinder the performance of machine learning algorithms because of the “curse of dimensionality.” Here, we attempted to overcome this potential limitation by employing feature selection, specifically a custom LASSO-based framework. We tested our approach by performing classification tasks on two datasets, one consisting of microbiome diversity data from poplar trees and one of gene presence/absence data from E. coli samples taken from hospitalized patients at risk of sepsis. Through a comparison to existing feature selection methods, we found that LASSO consistently outperformed on several key classification metrics, most notably AUC. On several prediction tasks, LASSO feature selection was able to increase AUC over the baseline method from around 0.5 to near-perfect values (0.99 and above). Our results suggest that our new LASSO framework can generate more meaningful features for classification algorithms relative to similar feature selection methods.