用固定文档池测试大模型重排行为，发现其多样性表现差异明显。
Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools
- 用固定8篇文档的新闻聚类，排除检索影响，专注重排策略差异。
- 大模型在小预算下词项覆盖差，大预算时一个增多样性，一个更冗余。
- 结果揭示大模型重排策略与传统方法本质不同，适合评估重排机制。
标准重排评估将排序行为与检索质量耦合，难以区分排名策略的影响。本文提出受控诊断方法，采用 Multi-News 聚类作为固定证据池，每池严格限定8篇文档，并对所有重排器使用相同输入。在此设定下，BM25 和 MMR 分别作为可解释的词汇匹配与多样性优化参考。在345个聚类上，我们发现：一个LLM在大选择预算下隐式增加多样性，另一个则加剧冗余；而多数情况下，LLM在小预算下词项覆盖显著低于基线。因此，LLM的排序结果与两个基准均存在显著差异，而非持续逼近任一策略。通过消除检索方差，可直接归因于排序策略本身。该诊断方法具有模型无关性，适用于任何重排器，包括开源系统和私有API。
原文摘要 · Abstract (English)
Standard reranking evaluations study how a reranker orders candidates returned by an upstream retriever. This setup couples ranking behavior with retrieval quality, so differences in output cannot be attributed to the ranking policy alone. We introduce a controlled diagnostic that isolates reranking by using Multi-News clusters as fixed evidence pools. We limit each pool to exactly eight documents and pass identical inputs to all rankers. Within this setup, BM25 and MMR serve as interpretable reference points for lexical matching and diversity optimization. Across 345 clusters, we find that redundancy patterns vary by model: one LLM implicitly diversifies at larger selection budgets, while another increases redundancy. In contrast, LLMs underperform on lexical coverage at small selection budgets. As a result, LLM rankings diverge substantially from both baselines rather than consistently approximating either strategy. By eliminating retrieval variance, we can attribute these differences directly to the ranking policy. This diagnostic is model-agnostic and applicable to any ranker, including open source systems and proprietary APIs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。