Conference Proceedings

Pairwise Crowd Judgments: Preference, Absolute, and Ratio

Ziying Yang, Alistair Moffat, Andrew Turpin

ADCS'18: PROCEEDINGS OF THE 23RD AUSTRALASIAN DOCUMENT COMPUTING SYMPOSIUM | ASSOC COMPUTING MACHINERY | Published : 2018

Abstract

Relevance judgments are conventionally formed by small numbers of experts using ordinal relevance scales defined by two or more relevance categories. Such judgments often contain many ties: documents in the same category that cannot be separated by relevance. Here we explore the use of crowd-sourcing and combined three-way relevance assessments using pairwise preference, absolute relevance, and relevance ratio, with forced choice testing and embedded quality control processes, seeking to reduce assessment ties, and to increase judgment consistency. In particular, the crowdsourced judgments from these three approaches were normalized into numeric relevance scores, and compared against judgmen..

View full abstract