On the Robustness of LLM Re-Rankings

Abstract

Large language models (LLMs) have become an indispensable part of document ranking systems.In this study we employ two datasets and two LLM models to explore the implications of relevant-item density in the input candidate list, and resilience to the ordering cues provided in the prompt. We found that rankings are consistent across density modes, but that re-ranking effectiveness is vulnerable to the ordering instructions provided in the prompt. In particular, in our experiments LLM re-rankers were much better at placing documents into ''most relevant first'' order than into ''most relevant last'' order. This difference has critical implications for practitioners.