Click-based evidence for decaying weight distributions in search effectiveness metrics
Yuye Zhang, Laurence AF Park, Alistair Moffat
INFORMATION RETRIEVAL | SPRINGER | Published : 2010
Search effectiveness metrics are used to evaluate the quality of the answer lists returned by search services, usually based on a set of relevance judgments. One plausible way of calculating an effectiveness score for a system run is to compute the inner-product of the run's relevance vector and a "utility" vector, where the ith element in the utility vector represents the relative benefit obtained by the user of the system if they encounter a relevant document at depth i in the ranking. This paper uses such a framework to examine the user behavior patterns-and hence utility weightings-that can be inferred from a web query log. We describe a process for extrapolating user observations from q..View full abstract
This work was supported by the Australian Research Council, Microsoft Research and NICTA. NICTA is funded by the Australian Government as represented by the Department of Broadband, Communications and the Digital Economy and the Australian Research Council through the ICT Center of Excellence program.