Machine Learning of Raman Spectroscopic Data: Comparison of Different Validation Strategies
David Lilek, Daniel Zimmermann, Lukas Steininger, Maurizio Musso, Bodo D. Wilts, Sonja Gamsjaeger, Daniel‐Ralph Hermann, Christoph Wiesner, Agnes Grünfelder, Birgit Herbinger, Katerina Prohaska
Abstract
ABSTRACT Machine learning (ML) techniques are valuable for analyzing complex biological SERS spectra, allowing for the detection of minor differences in cell composition. However, several challenges arise in the data analysis process, such as selecting the appropriate preprocessing methods, machine learning algorithms, and validation strategies to avoid under/overfitting and ensure reliable estimates. This study systematically compared various validation strategies and their impact on multiple ML classifiers using four biological datasets of varying complexities, in terms of class overlap, and sample variability. Therefore, a machine learning workflow was established, incorporating more than 10 classifiers and using nested cross‐validation (CV) for hyperparameter tuning and performance estimation. Five CV strategies were compared: Leave‐One‐Group‐Out, stratified K‐Fold, unstratified K‐Fold, Leave‐One‐Out, and nested CV. Our results demonstrate that stratified K‐Fold CV yielded performance nearly equivalent to nested CV in terms of accuracy and efficiency but with a reduced computational cost. Leave‐One‐Group‐Out strategy produced lower performance estimates than the other four methods, which may be more representative of real‐world performance. Conclusively, this work shows that simpler CV strategies can effectively replace computationally expensive nested CV in certain cases, while maintaining comparable performance. Nonetheless, careful consideration of overfitting remains crucial when employing these more efficient methods.