探究无标签策略能否消除选项顺序对大模型评测的影响
Accuracy and Order Sensitivity Diverge Under Label-Free Strategies
- 用生成匹配与独立评分两种方法消除选项顺序干扰
- 两种方法均未提升准确率，仅全选项+LLM匹配能保持基线表现
- 循环排列选项反而常提升性能，说明顺序影响未必有害
多选题评测广泛用于评估大语言模型，但得分同时反映知识水平与选项顺序敏感性，导致结果不可靠。本文测试在不显示选项标签的前提下让模型作答，是否可消除位置偏差并提升性能。我们评估了两种缓解偏见的策略：一是生成后匹配，二是独立评分（天然无位置依赖）。结果表明，两者均未显著提升准确率。完整分解显示，瓶颈在于隐藏选项本身，而非匹配步骤。唯一能稳定维持基线表现的配置是向模型展示所有选项并搭配一个大模型匹配器。然而，完全消除位置影响并未带来可靠准确率提升，而循环排列选项反而常能改善表现。对于两阶段提示，召回失衡的聚合度量和逐题的顺序敏感度量均无法有效实现去偏。
原文摘要 · Abstract (English)
Multiple-choice benchmarks are widely used to evaluate large language models, but MCQ scores conflate knowledge with sensitivity to option order, which makes them unreliable measures of model knowledge. In this paper, we test whether preventing a model from seeing option labels while committing to an answer removes positional influence and, in turn, improves performance. We evaluate two different strategies for mitigating bias. The first uses a generation-then-matching approach, and the second scores options in isolation, which is positionally unbiased by construction. Neither reliably improves accuracy. A complete decomposition shows that the bottleneck is withholding options, not the matching step. The only configuration that consistently matches the baseline is the one that shows the model all options paired with an LLM matcher. However, eliminating positional influence entirely still does not reliably yield accuracy gains, while cyclic permutation often improves them. For two-stage prompting, an aggregate measure of recall imbalance and a direct per-question measure of order sensitivity both fail to show reliable debiasing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。