Conference Proceedings

Judging relevance using magnitude estimation

E Maddalena, S Mizzaro, F Scholer, A Turpin

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) | SpringerLink | Published : 2015

Abstract

© Springer International Publishing Switzerland 2015. Magnitude estimation is a psychophysical scaling technique whereby numbers are assigned to stimuli to reflect the ratios of their perceived intensity. We report on a crowdsourcing experiment aimed at understanding if magnitude estimation can be used to gather reliable relevance judgements for documents, as is commonly required for test collection-based evaluation of information retrieval systems. Results on a small dataset show that: (i) magnitude estimation can produce relevance rankings that are consistent with more classical ordinal judgements; (ii) both an upper-bounded and an unbounded scale can be used effectively, though with some ..

View full abstract

University of Melbourne Researchers

Citation metrics