arXiv:2608.03091cs.IR2026-08中稿 · RecSys 2026

LLM重排器对候选顺序敏感,导致推荐结果不可靠。

Position Bias Undermines Preference Consistency in Listwise LLM-Based Reranking

  • 用等价排列测试偏好系统稳定性,发现位置偏差影响评分
  • 多模型多数据集实验显示偏好不一致与效果无关
  • 仅降低曝光偏倚无法保证重排结果一致性,适合关注公平性的研究者

大型语言模型(LLMs)作为推荐系统的列表级重排器展现出潜力,但其在候选对象不同排列下的可靠性尚不明确。由于推荐候选集本质上是无序的,重排器不应依赖序列化时的任意顺序。然而,解码器类LLM重排器可能使输入顺序影响模型得分、成对偏好和最终排名。本文研究位置偏差对基于LLM的重排过程的影响。不同于仅观察最终排序列表的变化,我们将不同候选排列生成的排序视为所诱导偏好系统的观测值。提出一种评估框架,用于度量成对偏好不稳定性、全局偏好不一致性和列表级输出一致性。该框架在成对、全局和输出三个层面刻画了候选顺序敏感性。跨多个LLM、数据集和列表长度的实验表明,这些一致性度量高度相关,但可与推荐有效性及边际位置曝光偏倚相分离。提升相关性或平衡各位置曝光,并不能必然恢复稳定的成对偏好、全局一致的偏好结构或一致的排序输出。结果表明,减少边际曝光偏差不足以建立基于LLM重排器的排名函数有效性。代码已开源:https://github.com/ejbito/InvariRank。

原文摘要 · Abstract (English)

Large language models (LLMs) have emerged as promising listwise rerankers for recommender systems, but their reliability under equivalent candidate permutations remains unclear. Since recommendation candidates form an unordered set, a reranker should not depend on the arbitrary order used to serialize them. However, decoder-only LLM rerankers can allow input order to affect model scores, pairwise preferences, and rankings. We study how position bias affects the ranking process induced by LLM-based rerankers. Instead of measuring only changes in final ranked lists, we treat rankings produced under equivalent candidate permutations as observations of an induced preference system. We introduce an evaluation framework measuring pairwise preference instability, global preference inconsistency, and listwise output consistency. This framework characterizes candidate-order sensitivity at the pairwise, global, and output levels. Experiments across multiple LLMs, datasets, and list lengths show that these consistency measures are closely aligned, but can diverge from recommendation effectiveness and marginal position-exposure bias. Improving relevance or flattening exposure across positions does not necessarily restore stable pairwise preferences, globally coherent preference structures, or consistent ranked outputs. These results show that reducing marginal exposure skew is insufficient to establish ranking-function validity in LLM-based reranking. Code is available at https://github.com/ejbito/InvariRank .

LLM重排位置偏差推荐系统偏好一致性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。