UQV100: A test collection with query variability
P Bailey, A Moffat, F Scholer, P Thomas
Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval - SIGIR '16 | Association for Computing Machinery (ACM) | Published : 2016
© 2016 ACM. We describe the UQV100 test collection, designed to incorporate variability from users. Information need "backstories" were written for 100 topics (or sub-topics) from the TREC 2013 and 2014 Web Tracks. Crowd workers were asked to read the backstories, and provide the queries they would use; plus effort estimates of how many useful documents they would have to read to satisfy the need. A total of 10,835 queries were collected from 263 workers. After normalization and spell-correction, 5,764 unique variations remained; these were then used to construct a document pool via Indri-BM25 over the ClueWeb12-B corpus. Qualified crowd workers made relevance judgments relative to the backs..View full abstract
Awarded by Australian Research Council
This work was supported by the Australian Research Council's Discovery Projects Scheme (project DP140102655). Xiaolu Lu (RMIT University) provided helpful assistance.