arXiv:2506.00643cs.CLcs.AI2025-06KDD被引 5

首个评估大模型多选题能力的基准,揭示其漏选严重问题。

SATA-BENCH: Select All That Apply Benchmark for Multiple Choice Questions

  • 构建SATA-BENCH多选题基准,覆盖阅读、法律、生物医学
  • 最强模型仅41.8%完全正确,暴露漏选与数量误判缺陷
  • 提出选择漏斗策略,提升准确率29%,推理成本降64%

大语言模型(LLMs)在单选题上评估日益普遍,但许多现实问题需从选项中选出所有正确答案,该能力仍被忽视。我们提出SATA-BENCH,首个专门用于评估大模型在多元领域(包括阅读理解、法律、生物医学)处理选择全部正确答案(SATA)问题的基准。对27个开源与专有模型的评估显示显著差距:最强模型仅达41.8%精确匹配,暴露模型无法可靠识别所有正确答案的问题。研究发现两大核心挑战:选择偏差——模型偏好某些选项,与内容无关;数量偏差——模型难以预测正确答案数量。为此,我们提出选择漏斗(Choice Funnel)解码策略,结合令牌去偏与自适应阈值,引导模型完成准确且完整的答案选择。该方法相比基线最高提升29%精确匹配,同时推理成本降低超64%。研究揭示当前大模型在多答案推理中的根本局限,并提供诊断与改进框架。我们公开SATA-BENCH与Choice Funnel,以推动大模型在真实多答案场景下的稳健决策发展。

原文摘要 · Abstract (English)

Large language models (LLMs) are increasingly evaluated on single-answer multiple-choice tasks, yet many real-world problems require identifying all correct answers from a set of options. This capability remains underexplored. We introduce SATA-BENCH, the first dedicated benchmark for evaluating LLMs on Select All That Apply (SATA) questions across diverse domains, including reading comprehension, law, and biomedicine. Our evaluation of 27 open-source and proprietary models reveals a significant gap: even the strongest model achieves only 41.8% exact match, exposing LLMs' inability to reliably identify all correct answers. We find that this weakness stems from two core challenges: selection bias - models favor certain choices regardless of content, and count bias - models fail to predict the correct number of answers. To address these issues, we propose Choice Funnel, a decoding strategy that combines token debiasing with adaptive thresholding to guide models toward complete and accurate selections. Choice Funnel achieves up to 29% higher exact match than competitive baselines while reducing inference cost by over 64%. Our findings expose fundamental limitations in current LLMs and introduce a new framework for diagnosing and improving multi-answer reasoning. We release SATA-BENCH and Choice Funnel to promote LLM development for robust decision-making in realistic, multi-answer applications.

多选题大模型评估推理能力基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。