Journal article
Incorporating user expectations and behavior into the measurement of search effectiveness
A Moffat, P Bailey, F Scholer, P Thomas
ACM Transactions on Information Systems | ASSOC COMPUTING MACHINERY | Published : 2017
DOI: 10.1145/3052768
Abstract
Information retrieval systems aim to help users satisfy information needs. We argue that the goal of the person using the system, and the pattern of behavior that they exhibit as they proceed to attain that goal, should be incorporated into the methods and techniques used to evaluate the effectiveness of IR systems, so that the resulting effectiveness scores have a useful interpretation that corresponds to the users' search experience. In particular, we investigate the role of search task complexity, and show that it has a direct bearing on the number of relevant answer documents sought by users in response to an information need, suggesting that useful effectiveness metrics must be goal sen..
View full abstractGrants
Awarded by Australian Research Council's Discovery Projects Scheme
Funding Acknowledgements
This work was supported by the Australian Research Council's Discovery Projects Scheme (project DP140102655). This article includes work previously published by the authors in a sequence of conference papers. Sections 3, 5, and 6 are based on work presented at the 2015 SIGIR Conference on Research and Development in Information Retrieval [Bailey et al. 2015], as is part of Section 4; the majority of Section 7 was presented as a short paper at the 2015 CIKM Conference on Information and Knowledge Management [Moffat et al. 2015]; and part of Section 4 was presented as a short paper at the 2015 Australasian Document Computing Symposium [Moffat et al. 2015].