Honest leave‐one‐out cross‐validation for estimating post‐tuning generalization error
Boxiang Wang, Hui Zou
Abstract
Many machine learning models have tuning parameters to be determined by the training data, and cross‐validation (CV) is perhaps the most commonly used method for selecting tuning parameters. This work concerns the problem of estimating the generalization error of a CV‐tuned predictive model. We propose to use an honest leave‐one‐out cross‐validation framework to produce a nearly unbiased estimator of the post‐tuning generalization error. By using the kernel support vector machine and the kernel logistic regression as examples, we demonstrate that the honest leave‐one‐out cross‐validation has very competitive performance even when competing with the state‐of‐the‐art .632+ estimator.
Topics & Concepts
GeneralizationCross-validationEstimatorGeneralization errorKernel (algebra)Computer scienceSupport vector machineLogistic regressionKernel methodMachine learningMean squared prediction errorArtificial intelligenceStatisticsMathematicsArtificial neural networkCombinatoricsMathematical analysisNeural Networks and ApplicationsFace and Expression RecognitionStructural Health Monitoring Techniques