Improving and Simplifying Pattern Exploiting Training
Derek Tam, Rakesh R. Menon, Mohit Bansal, Shashank Srivastava, Colin Raffel
2021Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing33 citationsDOIOpen Access PDF
Abstract
Recently, pre-trained language models (LMs) have achieved strong performance when finetuned on difficult benchmarks like Super-GLUE. However, performance can suffer when there are very few labeled examples available for fine-tuning. Pattern Exploiting Training (PET) is a recent approach that leverages patterns for few-shot learning. However, PET uses task-specific unlabeled data. In this paper, we focus on few shot learning without any unlabeled data and introduce ADAPET, which modifies PET's objective to provide denser supervision during fine-tuning. As a result, ADAPET outperforms PET on Su-perGLUE without any task-specific unlabeled data.
Topics & Concepts
Computer scienceTask (project management)Labeled dataCode (set theory)Focus (optics)Training setArtificial intelligenceShot (pellet)One shotTraining (meteorology)Machine learningProgramming languageMechanical engineeringMeteorologyEconomicsChemistryOpticsEngineeringOrganic chemistrySet (abstract data type)ManagementPhysicsTopic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications