Does Form Affect Function? An Extended Study of LLM Re-Ranking Behavior

Abstract

Large language models (LLMs) are used as re-rankers in multi-stage retrieval, refining document orderings to enhance performance. We first reproduce the re-ranking study of Sun et al. (EMNLP, 2023), evaluating supervised and LLM-based re-rankers on TREC-DL and BEIR benchmarks using a sliding-window permutation procedure. Our reproduced effectiveness values are within a 10% margin of Sun et al.'s reported results and validate their conclusion that LLM-based sliding window re-ranking improves effectiveness. We then extend the study by exploring two further dimensions, varying the first phase model, and the number of documents provided for re-ranking. Our results show that different first phase..