arXiv:2608.26762cs.CLcs.IR2026-08

同一排序质量的评分器,排序顺序不同会导致决策差异。

Equal Ranking Quality, Different Decisions: Training Order-Consistent LLM Scorers

  • 通过OC-SFT训练使评分器结果不受输入顺序影响
  • 相同nDCG@10下,重排后保留集重叠率仅0.66-0.84
  • 适合关注实际决策稳定性的模型部署者

重排序器、奖励模型和多文档问答评分器在单个LLM提示中对候选内容进行评分,其得分受输入顺序影响。尽管这些评分器按排序质量选择,但它们的输出决定具体决策:如阈值保留哪些项、读者回答什么问题或偏好模型选择什么。然而,相同排名质量并不保证相同决策——在段落重排序任务中,五个nDCG@10相差不超过0.010的评分器,在重新排序后保留集合的重叠率仅为0.66至0.84;一个已发表的重排序器虽在比较中取得最高保留集F1,重叠率仍仅为0.667。我们测试的所有提示时调整方法均无法消除这种顺序依赖性;唯一提升排序质量的方法也未改变三种决策。为此提出顺序一致性监督微调(OC-SFT),训练过程中使候选得分不依赖输入顺序,既保持排序质量,又显著提升决策稳定性:在三项任务上,所有决策稳定度指标均优于其他三个目标函数,其中对读者答案的翻转比例降至0.125(对比0.149–0.164),且在12个基础模型上优于顺序平均蒸馏。单一OC-SFT排列保留集重叠度超过十个现成平均排列。因此,评估应报告阈值保留内容及读者回答,而不仅是排序质量。代码开源:https://github.com/thomsonreuters/presentation-dependence。

原文摘要 · Abstract (English)

Rerankers, reward models and multi-document QA scorers score candidate documents or responses in one LLM prompt, so each score depends on their order. Such scorers are selected on ranking quality, but their scores determine a decision: what a score threshold retains, a reader answers, or a preference model selects. However, equal ranking quality does not imply equal decisions: on passage reranking, five trained scorers within 0.010 nDCG@10 retain sets that overlap by only 0.66-0.84 when reordered. A published reranker takes the highest retained-set F1 in our comparison and still overlaps by only 0.667. No prompt-time change we test removes that order dependence: the only one that gains ranking quality leaves all three decisions unchanged. Order-consistency SFT (OC-SFT) attenuates it in the weights, training a candidate's score not to depend on the order. It holds ranking quality and leads every decision-stability measure among trained scorers on all three tasks: it flips the reader's answer on 0.125 of permutation pairs against 0.149-0.164 for three other objectives that target order. It is more stable than order-averaged distillation on 12 base models, and one OC-SFT permutation retains sets that overlap more than ten averaged off-the-shelf permutations. A comparison should therefore report what a threshold retains and a reader answers, not ranking quality alone. Code is available at https://github.com/thomsonreuters/presentation-dependence.

评分器决策稳定排序质量

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。