arXiv:2506.01341cs.CL2025-06EMNLP被引 13

构建交互式猜谜基准,测试大模型多轮多步推理能力

TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Models

  • 设计类图灵机游戏的互动猜谜任务,模拟真实推理过程
  • 最佳模型在简单模式下准确率84%,复杂模式仅18%
  • 适合研究多轮推理、自洽性与反馈学习的学者使用

尽管大型语言模型取得显著进展,现有基准大多聚焦单轮或单步任务,难以反映真实场景中所需的迭代推理。为此,我们提出TurnBench,一个基于“图灵机棋盘游戏”启发的新型基准,通过交互式代码破解任务评估多轮、多步推理能力。每轮中,模型需通过连续猜测揭示隐藏的逻辑或算术规则,获得结构化反馈,并整合多轮线索。该动态设置要求模型随时间推演、根据历史信息调整策略并保持步骤一致性——这些能力在当前基准中尚未充分考察。TurnBench包含两种模式:Classic(标准推理)与Nightmare(更高复杂度,需强推理链)。为支持细粒度分析,我们提供中间推理步骤的真实标注。对主流LLM的评估显示:最佳模型在Classic模式下准确率达84%,但在Nightmare模式下降至18%;而人类参与者在两者中均达到100%。通过引入反馈环和隐藏规则,TurnBench降低数据污染风险,为诊断与提升大模型多步、多轮推理能力提供严谨测试平台。

原文摘要 · Abstract (English)

Despite impressive advances in large language models (LLMs), existing benchmarks often focus on single-turn or single-step tasks, failing to capture the kind of iterative reasoning required in real-world settings. To address this limitation, we introduce TurnBench, a novel benchmark that evaluates multi-turn, multi-step reasoning through an interactive code-breaking task inspired by the "Turing Machine Board Game." In each episode, a model must uncover hidden logical or arithmetic rules by making sequential guesses, receiving structured feedback, and integrating clues across multiple rounds. This dynamic setup requires models to reason over time, adapt based on past information, and maintain consistency across steps-capabilities underexplored in current benchmarks. TurnBench includes two modes: Classic, which tests standard reasoning, and Nightmare, which introduces increased complexity and requires robust inferential chains. To support fine-grained analysis, we provide ground-truth annotations for intermediate reasoning steps. Our evaluation of state-of-the-art LLMs reveals significant gaps: the best model achieves 84% accuracy in Classic mode, but performance drops to 18% in Nightmare mode. In contrast, human participants achieve 100% in both, underscoring the challenge TurnBench poses to current models. By incorporating feedback loops and hiding task rules, TurnBench reduces contamination risks and provides a rigorous testbed for diagnosing and advancing multi-step, multi-turn reasoning in LLMs.

多轮推理基准测试逻辑推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。