Journal article

From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442)

Nicola Ferro, Norbert Fuhr, Gregory Grefenstette, Joseph A Konstan, Pablo Castells, Elizabeth M Daly, Thierry Declerck, Michael D Ekstrand, Werner Geyer, Julio Gonzalo, Tsvi Kuflik, Krister Linden, Bernardo Magnini, Jian-Yun Nie, Raffaele Perego, Shapira Bracha, Ian Soboroff, Nava Tintarev, Verspoor Karin, Martijn C Willemsen Show all

Dagstuhl Manifestos | Schloss Dagstuhl | Published : 2018


We describe the state-of-the-art in performance modeling and prediction for Information Retrieval (IR), Natural Language Processing (NLP) and Recommender Systems (RecSys) along with its shortcomings and strengths. We present a framework for further research, identifying five major problem areas: understanding measures, performance analysis, making underlying assumptions explicit, identifying application features determining performance, and the development of prediction models describing the relationship between assumptions, features and resulting performance.

University of Melbourne Researchers