Handling Ties in Rank-Biased Measurement: Two Complementary Views

Abstract

Information retrieval systems often generate observations that are either finite unordered sets of documents, or finite prefixes of arbitrarily long ordered rankings of documents. To assess system quality those observations are then compared to corresponding reference sets or rankings to generate scores that measure how closely the observation achieves the target established by the reference. Recent work by Corsi and Urbano has considered Rank-Biased Overlap, a ''ranking against ranking'' measurement, and proposed an ''expectation over permutations'' mechanism to handle ties in rankings, where ties are groups of documents that are assigned the same importance. In this paper we note that ties..