Conference Proceedings

Relatively relevant: Assessor shift in document judgements

M Sanderson, F Scholer, A Turpin

ADCS 2010 - Proceedings of the Fifteenth Australasian Document Computing Symposium | Published : 2010


The evaluation of information retrieval systems relies on relevance judgements - human assessments of whether a document is relevant to a specified search request. In the past, it was demonstrated that test collection assessors disagree with each other to some extent on the relevance of documents and can be inconsistent in themselves. This paper describes a series of investigations on assessor consistency, which demonstrate that the inconsistency of an assessor varies over time. We show that when documents are presented to assessors in a relevance independent order, documents judged as relevant appear to cluster. Examining pairs of documents in a sequence ordered by timeof-judgement, we find..

View full abstract

University of Melbourne Researchers

Citation metrics