大模型评估中存在偏好热门选项的倾向，本文提出方法有效纠正此偏差。
Large Language Models Systematically Favor Popular Options: Evidence and Mitigation Across MCQs
- 构建可控基准PopMCQ，隔离并验证答案流行度对模型选择的影响
- 在极端情况下，模型选错热门选项比例高达66%，且自信度仍很高
- 提出无训练、轻量级推理修正方法PopDebias，适配各类开源大模型
多项选择题（MCQ）是评估大型语言模型（LLM）的标准形式，但选项的流行度可能干扰评估结果。现代大模型系统性地偏好更流行的错误选项，而非较少流行但正确的选项，这种现象称为“流行度偏差”。该模式与置信度误校准一致：当流行选项的准确率下降时，模型置信度依然保持高位。为系统性地分离这一现象，我们引入了PopMCQ基准，包含六种受控策略，通过改变选项流行度而固定正确答案。在最不利设置下（所有干扰项都比正确选项更受欢迎），模型选择错误流行选项的比例达到66%。为此，我们提出一种轻量级推理阶段修正方法PopDebias，通过估计并移除模型预测中的流行度先验来缓解该偏差。该方法无需微调，测试时无需标签（仅需少量校准集用于参数拟合），计算开销极低。在22个开源大模型（0.5B至32B参数）上的实验显示，其性能显著提升，最强压力下准确率最高提升达54.1个百分点。代码与数据已公开于https://github.com/DataScienceUIBK/PopMCQ。
原文摘要 · Abstract (English)
Multiple-choice questions (MCQs) are a standard format for evaluating large language models (LLMs), yet the popularity of answer options can confound evaluation. Modern LLMs systematically prefer popular but incorrect options over less popular correct ones, a vulnerability we call \textbf{popularity bias}. This pattern aligns with confidence miscalibration: model confidence remains high even as accuracy collapses for popular options. To systematically isolate this phenomenon, we introduce \textbf{PopMCQ}, a benchmark with six controlled strategies that vary option popularity while keeping the correct answer fixed. In our most adversarial setting, where all distractors are more popular than the correct option, models choose popular but wrong answers 66\% of the time. To mitigate this bias, we propose \textbf{PopDebias}, a lightweight inference-time correction that estimates and removes a popularity prior from model predictions. It requires no fine-tuning, is label-free at test time (using only a small calibration split for parameter fitting), and adds negligible computational cost. Experiments on 22 open-source LLMs (0.5B to 32B parameters) show consistent improvements, with accuracy gains up to 54.1 percentage points under strong popularity pressure. The code and data are available https://github.com/DataScienceUIBK/PopMCQ
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。