Conference Proceedings

Metrics, user models, and satisfaction

AF Wicaksono, A Moffat

WSDM 2020 - Proceedings of the 13th International Conference on Web Search and Data Mining | ACM | Published : 2020

Abstract

User satisfaction is an important factor when evaluating search systems, and hence a good metric should give rise to scores that have a strong positive correlation with user satisfaction ratings. A metric should also correspond to a plausible user model, and hence provide a tangible manifestation of how users interact with search rankings. Recent work has focused on metrics whose user models accurately portray the behavior of search engine users. Here we investigate whether those same metrics then also correlate with user satisfaction. We carry out experiments using various classes of metrics, and confirm through the lens of the C/W/L framework that the metrics with user models that reflect ..

View full abstract

Grants

Awarded by Australian Research Council


Funding Acknowledgements

We gratefully acknowledge the generosity of the creators of the datasets we have used, for making them available; and Paul Thomas (Microsoft) in particular for his assistance. This work was supported by the Australian Research Council's Linkage Projects scheme (project number LP150100252), by the Australian Research Council's Discovery Project Scheme (project number DP190101113), and by Seek.com.