Non-Clicks Mean Irrelevant? Propensity Ratio Scoring As a Correction
Nan Wang, Zhen Qin, Xuanhui Wang, Hongning Wang
Abstract
Recent advances in unbiased learning to rank (LTR) count on Inverse Propensity Scoring (IPS) to eliminate bias in implicit feedback. Though theoretically sound in correcting the bias introduced by treating clicked documents as relevant, IPS ignores the bias caused by (implicitly) treating non-clicked ones as irrelevant. In this work, we first rigorously prove that such use of click data leads to unnecessary pairwise comparisons between relevant documents, which prevent unbiased ranker optimization.
Topics & Concepts
Computer sciencePairwise comparisonPropensity score matchingWeightingSet (abstract data type)Rank (graph theory)Machine learningArtificial intelligenceStatisticsMathematicsProgramming languageCombinatoricsMedicineRadiologyAdvanced Image and Video Retrieval TechniquesMachine Learning and AlgorithmsDomain Adaptation and Few-Shot Learning