Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data
Lingkai Kong, Haoming Jiang, Yuchen Zhuang, Jie Lyu, Tuo Zhao, Chao Zhang
Abstract
Fine-tuned pre-trained language models can suffer from severe miscalibration for both in-distribution and out-of-distribution (OOD) data due to over-parameterization. To mitigate this issue, we propose a regularized fine-tuning method.
Topics & Concepts
Regularization (linguistics)Manifold (fluid mechanics)Computer scienceCalibrationInterpolation (computer graphics)SmoothnessDistribution (mathematics)Data pointManifold alignmentArtificial intelligenceAlgorithmNonlinear dimensionality reductionMathematicsStatisticsEngineeringMotion (physics)Mathematical analysisDimensionality reductionMechanical engineeringTopic ModelingTime Series Analysis and ForecastingAnomaly Detection Techniques and Applications