arXiv:2608.29632cs.SEcs.AI2026-08中稿 · ICML

评测大模型在未知信息下的编程交互能力,发现现有模型表现有限。

InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information

论文配图:InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information
图 1 · 摘自论文原文
  • 构建322道交互式编程题库,支持离线评估动态信息获取与状态追踪
  • 顶尖模型在交互任务上成功率不足,暴露出协议违规和查询超限问题
  • 提出细粒度错误分类体系,帮助定位模型短板,适合算法推理研究者使用

竞赛编程正被广泛用于评估大语言模型的算法推理能力。然而,现有基准主要聚焦于信息完全公开的任务,忽略了算法推理中关键的一环:模型生成代码在关键信息未提前揭示时仍能有效运行的能力。交互式问题作为竞赛编程的重要组成部分,要求程序在严格协议约束下与裁判程序(interactor)进行多轮交互,仅在提交查询后才获得新信息,且受限于查询次数。为填补这一空白,我们提出InteractBench,包含从Codeforces、AtCoder、IOI和ICPC精选的322道高质量交互式题目,并配备可执行本地interactor,支持完全离线评估。与现有基准不同,InteractBench关注模型生成代码是否具备动态获取信息与维护状态的能力。评估结果显示显著的交互差距:即使最先进的推理模型在交互任务上的成功表现依然有限。除算法逻辑错误外,协议违规和查询预算超限也频繁发生。我们进一步提出细粒度失败分类体系,以诊断根本原因。代码已开源。

原文摘要 · Abstract (English)

Competitive programming is increasingly being used to evaluate the algorithmic reasoning capabilities of large language models (LLMs). However, existing benchmarks primarily focus on full-information tasks where all problem inputs are provided upfront. This overlooks a critical dimension of algorithmic reasoning: the ability of generated programs to operate when key information is not revealed upfront. Interactive problems, a distinctive component of competitive programming, embody this challenge. These problems require programs to engage in multi-round interaction with an interactor (a judge program) under strict protocol constraints and limited query budgets, with new information revealed only in response to queries. To address this gap, we introduce InteractBench, a benchmark comprising 322 high-quality interactive problems curated from Codeforces, AtCoder, IOI, and ICPC. Each problem is packaged with executable local interactors, enabling fully offline evaluation. Unlike existing benchmarks, InteractBench assesses whether model-generated code can acquire information and track state dynamically. Our evaluation reveals a significant interaction gap: even the most advanced reasoning models achieve limited success on interactive problems. Beyond success rates, we propose a fine-grained failure taxonomy to diagnose the root causes of these deficiencies. Although algorithmic logic errors remain dominant, protocol violations and query-budget overruns are frequent. Code is available at https://github.com/kmsgk0/InteractBench.

算法推理交互式编程大模型评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。