arXiv:2507.17788cs.LGcs.AI2025-07被引 3

动态调整重复次数,降低大模型排序时的位置偏差与计算开销。

Adaptive Repetition for Mitigating Position Bias in LLM-Based Ranking

  • 根据每个实例的反馈动态决定重复调用次数,避免固定重复浪费算力。
  • 相比静态重复,平均减少81%~87%的LLM调用,准确率基本保持不变。
  • 适合需要高一致性排序的场景,如推荐系统、答案对齐评估。

使用大语言模型(LLM)基于给定标准对候选项进行排序或评估时,候选项在提示中的顺序会影响模型最终决策,这种现象称为位置偏差。已有研究表明,即使在大型模型中该偏差依然存在,且严重程度随模型和任务而异。此外,LLM还表现出不同程度的低重复一致性:相同排序下多次调用可能产生不同结果。为缓解这两类不一致性,常见做法是多次以不同顺序调用模型并采用多数投票聚合结果,但显著增加计算成本。本文发现,位置偏差的方向和强度在单个数据集内差异显著,因此需针对每个实例设计自适应策略。为此,我们提出一种动态早停方法,按需确定每例所需的重复次数。在三个不同规模的LLM及重排序与对齐两项任务上评估,该方法平均减少81%的调用次数,同时保持准确率。进一步引入基于置信度的改进,平均减少87%调用,仅轻微牺牲精度。

原文摘要 · Abstract (English)

When using LLMs to rank items based on given criteria, or evaluate answers, the order of candidate items can influence the model's final decision. This sensitivity to item positioning in a LLM's prompt is known as position bias. Prior research shows that this bias exists even in large models, though its severity varies across models and tasks. In addition to position bias, LLMs also exhibit varying degrees of low repetition consistency, where repeating the LLM call with the same candidate ordering can lead to different rankings. To address both inconsistencies, a common approach is to prompt the model multiple times with different candidate orderings and aggregate the results via majority voting. However, this repetition strategy, significantly increases computational costs. Extending prior findings, we observe that both the direction -- favoring either the earlier or later candidate in the prompt -- and magnitude of position bias across instances vary substantially, even within a single dataset. This observation highlights the need for a per-instance mitigation strategy. To this end, we introduce a dynamic early-stopping method that adaptively determines the number of repetitions required for each instance. Evaluating our approach across three LLMs of varying sizes and on two tasks, namely re-ranking and alignment, we demonstrate that transitioning to a dynamic repetition strategy reduces the number of LLM calls by an average of 81%, while preserving the accuracy. Furthermore, we propose a confidence-based adaptation to our early-stopping method, reducing LLM calls by an average of 87% compared to static repetition, with only a slight accuracy trade-off relative to our original early-stopping method.

大模型排序位置偏差动态重复效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。