Conference Proceedings

Corpus Bootstrapping for Assessment of the Properties of Effectiveness Measures

J Zobel, L Rashidi

CIKM '20: Proceedings of the 29th ACM International Conference on Information & Knowledge Management | ACM | Published : 2020


Bootstrapping is an established tool for examining the behaviour of offline information retrieval (IR) experiments, where it has primarily been used to assess statistical significance and the robustness of significance tests. In this work we consider how bootstrapping can be used to assess the reliability of effectiveness measures for experimental IR. We use bootstrapping of the corpus of documents rather than, as in most prior work, the set of queries. We demonstrate that bootstrapping can provide new insights into the behaviour of effectiveness measures: the precision of the measurement of a system for a query can be quantified; some measures are more consistent than others; rankings of sy..

View full abstract

Citation metrics