Optimizing Machine Learning Models with Data-level Approximate Computing: The Role of Diverse Sampling, Precision Scaling, Quantization and Feature Selection Strategies

Abstract

• Dataset Preparation : We replaced unknown values with nearest instance values based on the same or other features, avoiding zero substitution. • Feature Selection : The modified K-tree approach reduces computations, overfitting, and training time, enhancing KNN and SVM accuracy. • Sampling Strategies : Four sampling methods—random, reservoir, stratified, and systematic—were evaluated to improve learning efficiency for SVM and KNN. • Quantization Techniques : Uniform and µ-law non-uniform quantization were used to lower data precision and significantly reduce computational complexity. • Precision Scaling : We employed fixed-point formats and reduced bit counts to minimize computational resources while maintaining accuracy. • Medical Application : Approximate computing was applied to breast cancer data, cutting power usage and computational load while preserving accuracy for diagnostics. Efficiency, low-power consumption, and real-time processing in embedded machine learning implementations are critical, particularly for models deployed in environments with large-scale data processing and resource-constrained environments. This paper investigates the application of approximate computing techniques as a viable solution to reduce computational complexity and optimize machine learning models, focusing on two widely used supervised machine learning models: k-nearest neighbors (KNN) and support vector machines (SVM). Although many studies compare machine learning classification techniques, the combined use of optimization strategies remains underexplored. Specifically, the combined utilization of feature selection, sampling, quantization, precision scaling, and relaxation methods for the purpose of optimizing and acquiring training and validation data is underexplored, particularly within the context of medical diagnosis datasets. In this paper, we propose a framework that uses data-level approximate computing techniques, including by diverse sampling strategies, precision scaling, quantization, and feature selection methods, to evaluate the impact of these techniques on the computational efficiency and accuracy of KNN and SVM models. Experimental results demonstrate that with careful application of approximate computing strategies, especially in critical applications such as medical diagnosis, it is possible to achieve considerable gains in efficiency while maintaining acceptable levels of accuracy. The combined application of these methods by selecting 3 features and quantizing the data values to 8 levels, then applying random sampling with 30% reductions and scaling the precision at 5 bits resulted in reductions of 87.5% in computation, 76.9% in memory usage, and 17% in delay, without any degradation in accuracy, as validated by tenfold cross-validation, Training Data validation, and full dataset validation. This study confirms the potential of approximate computing to optimize machine learning workflows, making it particularly suitable for applications with limited computational resources. The source code is publicly available online https://github.com/AyadMDalloo/DatalvlAxC .

Topics & Concepts