Extending Label Aggregation Models with a Gaussian Process to Denoise Crowdsourcing Labels
Dan Li, Maarten de Rijke
Abstract
Label aggregation (LA) is the task of inferring a high-quality label for an example from multiple noisy labels generated by either human annotators or model predictions. Existing work on LA assumes a label generation process and designs a probabilistic graphical model (PGM) to learn latent true labels from observed crowd labels. However, the performance of PGM-based LA models is easily affected by the noise of the crowd labels. As a consequence, the performance of LA models differs on different datasets and no single LA model outperforms the rest on all datasets.
Topics & Concepts
CrowdsourcingComputer scienceGraphical modelProbabilistic logicArtificial intelligenceProcess (computing)Machine learningNoise (video)Gaussian processTask (project management)Statistical modelQuality (philosophy)Pattern recognition (psychology)Data miningGaussianImage (mathematics)Quantum mechanicsWorld Wide WebEpistemologyPhilosophyManagementEconomicsOperating systemPhysicsMobile Crowdsensing and CrowdsourcingAnomaly Detection Techniques and ApplicationsData Stream Mining Techniques