自动构建代码修复任务，评估大模型系统级推理能力
Breakpoint: Scalable evaluation of system-level reasoning in LLM code agents
- 用对抗性污染真实代码库生成修复任务
- 可调控任务难度，最难题目零成功率
- 适合评估代码智能体在复杂场景的表现
现有大语言模型评测多聚焦短期、局部推理。长期任务基准（如 SWE-bench）依赖人工标注问题，扩展困难且成本高，评估易饱和。但现实任务如软件工程需快速理解并动态操作新颖复杂结构。我们提出 Breakpoint，通过对抗性污染真实软件仓库中的函数，自动生成代码修复任务。该方法系统控制两个维度的难度：局部推理（以环路复杂度等代码复杂度指标衡量）与系统级推理（以调用图中心性和同时被污染的相互依赖函数数量衡量）。在超过 900 个生成任务上的实验表明，该方法可无限扩展难度，当前顶尖模型在最简单任务上成功率 55%，在最难任务上降至 0%。
原文摘要 · Abstract (English)
Benchmarks for large language models (LLMs) have predominantly assessed short-horizon, localized reasoning. Existing long-horizon suites (e.g. SWE-bench) rely on manually curated issues, so expanding or tuning difficulty demands expensive human effort and evaluations quickly saturate. However, many real-world tasks, such as software engineering or scientific research, require agents to rapidly comprehend and manipulate novel, complex structures dynamically; evaluating these capabilities requires the ability to construct large and varied sets of problems for agents to solve. We introduce Breakpoint, a benchmarking methodology that automatically generates code-repair tasks by adversarially corrupting functions within real-world software repositories. Breakpoint systematically controls task difficulty along two clear dimensions: local reasoning (characterized by code complexity metrics such as cyclomatic complexity) and system-level reasoning (characterized by call-graph centrality and the number of simultaneously corrupted interdependent functions). In experiments across more than 900 generated tasks we demonstrate that our methodology can scale to arbitrary difficulty, with state-of-the-art models' success rates ranging from 55% on the easiest tasks down to 0% on the hardest.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。