研究大模型在二选一问答中因选项顺序产生的偏好偏差，发现不确定性越高，偏差越严重。
Positional Bias in Binary Question Answering: How Uncertainty Shapes Model Preferences
- 通过调整数据上下文和答案相关性，构建从低到高不确定性的测试集。
- 在高不确定性下，模型对选项位置的依赖呈指数增长，正确答案被选中的概率显著受顺序影响。
- 适用于评估大模型在模糊判断场景下的公平性，适合关注AI决策偏见的研究者。
二选一问答中的位置偏差指模型仅因选项排列顺序而系统性偏好某一选项。本研究量化并分析了五种大型语言模型在不同答案不确定性条件下的位置偏差。通过在SQuAD-it数据集基础上增加错误选项，并逐步减少上下文信息、增加无关答案，构建出从低到高不确定性的多个版本数据集。同时，评估两个天然高不确定性的基准：(1) WebGPT——人类评分质量不一致的问题对；(2) Winning Arguments——预测Reddit r/ChangeMyView中更具说服力的观点。在每个数据集上，将正确（或高质量/有说服力）选项交替置于第1或第2位，计算偏好公平性（Preference Fairness）与位置一致性（Position Consistency）。结果表明，在低不确定性条件下位置偏差几乎不存在，但当判断正确答案变得困难时，偏差呈指数级增长。
原文摘要 · Abstract (English)
Positional bias in binary question answering occurs when a model systematically favors one choice over another based solely on the ordering of presented options. In this study, we quantify and analyze positional bias across five large language models under varying degrees of answer uncertainty. We re-adapted the SQuAD-it dataset by adding an extra incorrect answer option and then created multiple versions with progressively less context and more out-of-context answers, yielding datasets that range from low to high uncertainty. Additionally, we evaluate two naturally higher-uncertainty benchmarks: (1) WebGPT - question pairs with unequal human-assigned quality scores, and (2) Winning Arguments - where models predict the more persuasive argument in Reddit's r/ChangeMyView exchanges. Across each dataset, the order of the "correct" (or higher-quality/persuasive) option is systematically flipped (first placed in position 1, then in position 2) to compute both Preference Fairness and Position Consistency. We observe that positional bias is nearly absent under low-uncertainty conditions, but grows exponentially when it becomes doubtful to decide which option is correct.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。