Crowd Worker Strategies in Relevance Judgment Tasks
Lei Han, Eddy Maddalena, Alessandro Checco, Cristina Sarasua, Ujwal Gadiraju, Kevin Roitero, Gianluca Demartini
Abstract
Crowdsourcing is a popular technique to collect large amounts of human-generated labels, such as relevance judgments used to create information retrieval (IR) evaluation collections. Previous research has shown how collecting high quality labels from a crowdsourcing platform can be challenging. Existing quality assurance techniques focus on answer aggregation or on the use of gold questions where ground-truth data allows to check for the quality of the responses.
Topics & Concepts
CrowdsourcingRelevance (law)Computer scienceFocus (optics)Quality (philosophy)Ground truthData scienceQuality assuranceInformation retrievalWorld Wide WebArtificial intelligenceEngineeringPhilosophyEpistemologyPhysicsLawPolitical scienceExternal quality assessmentOperations managementOpticsMobile Crowdsensing and CrowdsourcingOpen Source Software InnovationsPersonal Information Management and User Behavior