Conference Proceedings

Bayesian Inferential Risk Evaluation On Multiple IR Systems

Rodger Benham, Ben Carterette, J Shane Culpepper, Alistair Moffat

Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval | ACM | Published : 2020

Abstract

Information retrieval (IR) ranking models in production systems continually evolve in response to user feedback, insights from research, and new developments. Rather than investing all engineering resources to produce a single challenger to the existing system, a commercial provider might choose to explore multiple new ranking models simultaneously. However, even small changes to a complex model can have unintended consequences. In particular, the per-topic effectiveness profile is likely to change, and even when an overall improvement is achieved, gains are rarely observed for every query, introducing the risk that some users or queries may be negatively impacted by the new model if deploye..

View full abstract

Grants

Awarded by Australian Research Council


Funding Acknowledgements

The first author was supported by an RMIT Vice Chancellor's PhD Scholarship. This work was also partially supported by the Australian Research Council's Discovery Projects funding scheme (grant DP190101113).