How to Measure the Reproducibility of System-oriented IR Experiments
Timo Breuer, Nicola Ferro, Norbert Fuhr, Maria Maistro, Tetsuya Sakai, Philipp Schaer, Ian Soboroff
Abstract
Replicability and reproducibility of experimental results are primary concerns in all the areas of science and IR is not an exception. Besides the problem of moving the field towards more reproducible experimental practices and protocols, we also face a severe methodological issue: we do not have any means to assess when reproduced is reproduced. Moreover, we lack any reproducibility-oriented dataset, which would allow us to develop such methods.
Topics & Concepts
ReproducibilityMeasure (data warehouse)Field (mathematics)Computer scienceArtificial intelligenceFace (sociological concept)Materials sciencePattern recognition (psychology)Data miningReliability (semiconductor)MetrologySample (material)Biomedical engineeringMathematicsMachine Learning in Materials ScienceScientific Computing and Data ManagementExplainable Artificial Intelligence (XAI)