提出新评估框架SCOPE，消除大模型答题中的位置偏见。
SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Models
- 通过空提示反复测试，建模模型的位置偏好分布。
- 逆向调整选项位置，使正确答案随机选择概率均衡。
- 避免相似干扰项紧邻正确答案，阻断表面线索猜题。
大语言模型在多项选择任务中可能因选项位置或标签的固有偏见而获得虚高分数，而非真正理解内容。本文提出SCOPE评估框架，以数据无关方式测量并缓解此类选择偏见。通过多次调用无语义内容的空提示，SCOPE估计每个模型的独特位置偏见分布，并根据逆偏置分布重新分配答案位置，从而均衡‘幸运率’（即仅靠运气选对的概率）。此外，该方法阻止语义相近的干扰项与正确答案相邻，有效抑制基于表面邻近性的误判猜测。在多个基准测试中，SCOPE始终优于现有去偏方法，表现出更稳定的性能提升和更清晰的正确选项置信度分布。该框架为提升大模型评估的公平性与可靠性提供了新标准。
原文摘要 · Abstract (English)
Large Language Models (LLMs) can achieve inflated scores on multiple-choice tasks by exploiting inherent biases in option positions or labels, rather than demonstrating genuine understanding. This study introduces SCOPE, an evaluation framework designed to measure and mitigate such selection bias in a dataset-independent manner. By repeatedly invoking a null prompt that lacks semantic content, SCOPE estimates each model's unique position-bias distribution. It then redistributes the answer slot according to the inverse-bias distribution, thereby equalizing the lucky-rate, the probability of selecting the correct answer by chance. Furthermore, it prevents semantically similar distractors from being placed adjacent to the answer, thereby blocking near-miss guesses based on superficial proximity cues. Across multiple benchmark experiments, SCOPE consistently outperformed existing debiasing methods in terms of stable performance improvements and showed clearer confidence distributions over correct options. This framework thus offers a new standard for enhancing the fairness and reliability of LLM evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。