HawkEye: A robust loss function for regression with bounded, smooth, and insensitive zone characteristics
Mushir Akhtar, M. Tanveer, Mohd. Arshad
Abstract
Support vector regression (SVR) encounters challenges when confronted with outliers and noise, primarily due to the limitations of the traditional ɛ ɛ -insensitive loss function. To address this, bounded loss functions have gained traction for their robustness and improved generalization. More recent advancements, such as BLINEX and bounded least square loss, focus on smooth bounded loss functions that enable efficient gradient-based optimization. However, these approaches lack an insensitive zone, which is crucial for mitigating minor deviations and noise. The challenge of designing a loss function that combines boundedness , smoothness, and an insensitive zone remains unresolved in the current literature. To address this issue, we develop the HawkEye loss, a novel formulation that integrates boundedness , smoothness, and the presence of an insensitive zone. This unique combination enhances the robustness and generalization capabilities of SVR models, particularly in the presence of noise and outliers. Notably, the HawkEye loss is the first in SVR literature to simultaneously incorporate boundedness, smoothness, and an insensitive zone. Leveraging this breakthrough, we integrate the HawkEye loss into the least squares framework of SVR and yield a new robust and scalable model termed HE-LSSVR. The optimization problem inherent to HE-LSSVR is addressed by harnessing the adaptive moment estimation (Adam) algorithm, known for its adaptive learning rate and efficacy in handling large-scale problems. To our knowledge, this is the first time Adam has been employed to solve an SVR problem. To empirically validate the proposed HE-LSSVR model, we evaluate it on UCI, synthetic, time series, and brain age datasets. The experimental outcomes unequivocally reveal the superiority of the HE-LSSVR model both in terms of its remarkable generalization performance and its efficiency in training time. The code of the proposed model is publicly available at https://github.com/mtanveer1/HawkEye .