New Approaches to Interactive Sessional Search for Complex Tasks

A Flexible Resource for Top-Weighted Comparisons Between Sets and Rankings

A Moffat, J Mackenzie, A Mallia, M Petri

2025-01-01

We describe rbstar, a toolkit of software for carrying out measurements when the goal is to determine how similar a system observa..

Expert Finding Revisited: A Uniform Exploration of Methods

M Azimi, A Moffat, J Zobel

2025-01-01

The goal of expert search is to identify individuals with knowledge or skills in a specific field or for a specific problem. In th..

Handling Ties in Rank-Biased Measurement: Two Complementary Views

A Moffat

2025-01-01

Information retrieval systems often generate observations that are either finite unordered sets of documents, or finite prefixes o..

Refined Medical Search via Dense Retrieval and User Interaction

R Goli, A Moffat, G Buchanan

2025-01-01

Users formulate search queries that reflect an information need. Those queries are then submitted to a search service in the expec..

Rank-Biased Quality Measurement for Sets and Rankings

A Moffat, J MacKenzie, A Mallia, M Petri

2024-12-08

Experiments often result in the need to compare an observation against a reference, where observation and reference are selections..

Query Variability and Experimental Consistency: A Concerning Case Study

L Rashidi, J Zobel, A Moffat

2024-08-02

In offline experimentation, the effectiveness of a search engine is evaluated using a document collection, a set of queries agains..

How much freedom does an effectiveness metric really have?

A Moffat, J Mackenzie

2024-06-01

It is tempting to assume that because effectiveness metrics have free choice to assign scores to search engine result pages (SERPs..

User-oriented metrics for search engine deterministic sort orders

A Moffat

2024-01-01

A recent paper proposes the “buying power” (bp) metric for assessing the quality of the product rankings generated by e-commerce s..

The Impact of Judgment Variability on the Consistency of Offline Effectiveness Measures

L Rashidi, J Zobel, A Moffat

2023-08-18

Measurement of the effectiveness of search engines is often based on use of relevance judgments. It is well known that judgments c..

A Flexible Framework for Offline Effectiveness Metrics

A Moffat, J MacKenzie, P Thomas, L Azzopardi

2022-07-06

The use of offline effectiveness metrics is one of the cornerstones of evaluation in information retrieval. Static resources that ..

Can Users Predict Relative Query Effectiveness?

O Zendel, MP Ebrahim, JS Culpepper, A Moffat, F Scholer

2022-07-06

Any given information need can be expressed via a wide range of possible queries. Recent work with such query variations has demon..

Offline recommender system evaluation: Challenges and new directions

P Castells, A Moffat

2022-06-01

Offline evaluation is an essential complement to online experiments in the selection, improvement, tuning, and deployment of recom..

A Survey on Trajectory Data Management, Analytics, and Learning

S Wang, Z Bao, JS Culpepper, G Cong

2022-03-31

Recent advances in sensor and mobile devices have enabled an unprecedented increase in the availability and collection of urban tr..

Let Trajectories Speak Out the Traffic Bottlenecks

H Luo, Z Bao, G Cong, JS Culpepper, NLD Khoa

2022-02-01

Traffic bottlenecks are a set of road segments that have an unacceptable level of traffic caused by a poor balance between road ca..

Batch Evaluation Metrics in Information Retrieval: Measures, Scales, and Meaning

A Moffat

2022-01-01

A sequence of recent papers, including in this journal, has considered the role of measurement scales in information retrieval (IR..

Modality Effects When Simulating User Querying Tasks

J MacKenzie, A Moffat

2021-07-11

When faced with an information need, different users are likely to issue different queries. A renewed interest in these user query..

Modeling search and session effectiveness

AF Wicaksono, A Moffat

2021-07-01

Many information needs cannot be resolved with a single query, and instead lead naturally to a sequence of queries, issued as a se..

Different Keystrokes for Different Folks: Visualizing Crowdworker Querying Behavior

R Benham, J MacKenzie, JS Culpepper, A Moffat

2021-03-14

Search engine users retrieve relevant information for an information need using keyword queries. Different users may have similar ..

Bayesian System Inference on Shallow Pools

R Benham, A Moffat, JS Culpepper

2021-01-01

IR test collections make use of human annotated judgments. However, new systems that surface unjudged documents high in their resu..

CC-News-En: A Large English News Corpus

Joel Mackenzie, Rodger Benham, Matthias Petri, Johanne R Trippas, J Shane Culpepper, Alistair Moffat

2020-10-19

We describe a static, open-access news corpus using data from the Common Crawl Foundation, who provide free, publicly available we..

Bayesian Inferential Risk Evaluation On Multiple IR Systems

Rodger Benham, Ben Carterette, J Shane Culpepper, Alistair Moffat

2020-07-25

Information retrieval (IR) ranking models in production systems continually evolve in response to user feedback, insights from res..

Spatial Object Recommendation with Hints: When Spatial Granularity Matters

H Luo, J Zhou, Z Bao, S Li, JS Culpepper, H Ying, H Liu, H Xiong

2020-07-25

Existing spatial object recommendation algorithms generally treat objects identically when ranking them. However, spatial objects ..

On the Pluses and Minuses of Risk

R Benham, A Moffat, JS Culpepper

2020-02-27

Evaluating the effectiveness of retrieval models has been a mainstay in the IR community since its inception. Generally speaking, ..

Metrics, user models, and satisfaction

AF Wicaksono, A Moffat

2020-01-01

User satisfaction is an important factor when evaluating search systems, and hence a good metric should give rise to scores that h..

Taking Risks with Confidence

Rodger Benham, Ben Carterette, Alistair Moffat, J Shane Culpepper

2019-01-01

Risk-based evaluation is a failure analysis tool that can be combined with traditional effectiveness metrics to ensure that the im..