实测发现大模型在长代码上下文推理中表现极差,根本无法直接处理完整代码库。
The Limits of Long-Context Reasoning in Automated Bug Fixing
- 通过任务分解成短步骤提升修复成功率,而非真正利用长上下文。
- 64k超长上下文下模型修复率降至7%,甚至零成功。
- 适合关注模型真实推理能力、而非单纯参数量的开发者参考。
随着上下文长度快速增加,人们普遍认为大语言模型(LLMs)可直接对整个代码库进行推理。尽管近期模型在软件工程基准上表现强劲,尤其配合代理工作流时。本文系统评估当前LLMs在长上下文代码调试与补丁生成中的可靠性。基于SWE-bench Verified设置,使用mini-SWE-agent代理框架,性能显著提升:GPT-5-nano在100个样本上最高达成31%修复率,开源模型Deepseek-R1-0528也表现良好。但令牌级分析显示,成功路径通常控制在20k–30k令牌内,上下文越长,成功率越低,表明成功主要来自任务分解,而非长上下文推理。为直接测试长上下文能力,我们构建数据管道,通过人工扩增输入上下文至64k令牌(确保完美召回),研究单次补丁生成。结果性能急剧下降:Qwen3-Coder-30B-A3B仅7%修复率,而GPT-5-nano全失败。定性分析揭示系统性失败模式,如虚构补丁、错误文件目标、格式错误的补丁头。总体表明,当前模型名义上下文长度与其实际可用容量间存在巨大差距,现有代理编码基准并未真正评估长上下文推理能力。
原文摘要 · Abstract (English)
Rapidly increasing context lengths have led to the assumption that large language models (LLMs) can directly reason over entire codebases. Concurrently, recent advances in LLMs have enabled strong performance on software engineering benchmarks, particularly when paired with agentic workflows. In this work, we systematically evaluate whether current LLMs can reliably perform long-context code debugging and patch generation. Using SWE-bench Verified as a controlled experimental setting, we first evaluate state-of-the-art models within an agentic harness (mini-SWE-agent), where performance improves substantially: GPT-5-nano achieves up to a 31\% resolve rate on 100 samples, and open-source models such as Deepseek-R1-0528 obtain competitive results. However, token-level analysis shows that successful agentic trajectories typically remain under 20k-30k tokens, and that longer accumulated contexts correlate with lower success rates, indicating that agentic success primarily arises from task decomposition into short-context steps rather than effective long-context reasoning. To directly test long-context capability, we construct a data pipeline where we artificially inflate the context length of the input by placing the relevant files into the context (ensuring perfect retrieval recall); we then study single-shot patch generation under genuinely long contexts (64k tokens). Despite this setup, performance degrades sharply: Qwen3-Coder-30B-A3B achieves only a 7\% resolve rate at 64k context, while GPT-5-nano solves none of the tasks. Qualitative analysis reveals systematic failure modes, including hallucinated diffs, incorrect file targets, and malformed patch headers. Overall, our findings highlight a significant gap between nominal context length and usable context capacity in current LLMs, and suggest that existing agentic coding benchmarks do not meaningfully evaluate long-context reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。