The Human Evaluation Datasheet: A Template for Recording Details of Human Evaluation Experiments in NLP
Anastasia Shimorina, Anja Belz
Abstract
This paper presents the Human Evaluation Datasheet (HEDS), a template for recording the details of individual human evaluation experiments in Natural Language Processing (NLP), and reports on first experience of researchers using HEDS sheets in practice. Originally taking inspiration from seminal papers by Bender and Friedman (2018), These are crucial for scientifically principled evaluation, but the overhead of completing a detailed datasheet is substantial, and we discuss possible ways of addressing this and other issues observed in practice.
Topics & Concepts
DatasheetComparabilityComputer scienceArtificial intelligenceOverhead (engineering)Data scienceMachine learningNatural language processingProgramming languageCombinatoricsMathematicsOperating systemTopic ModelingEthics and Social Impacts of AIExplainable Artificial Intelligence (XAI)