arXiv:2603.15921cs.SEcs.AI2026-03被引 4

首次系统评估大模型自诊断修复能力,发现其核心瓶颈是定位故障的推理能力。

VIBEPASS: Can Vibe Coders Really Pass the Vibe Check?

  • 设计双任务评测框架,分离测试生成与修复效果
  • 12个前沿模型中,90%能生成合法测试用例但难发现隐藏缺陷
  • 自动生成测试若失败反而让修复更差,凸显故障推理关键性

随着大语言模型推动编程向'氛围编码'演进,自主编程工具愈发依赖模型自我诊断与修复潜在缺陷——这一能力虽至关重要却从未被系统评估。本文提出 ame{},首个联合评估两个耦合任务的实证分解:故障触发测试生成(FT-Test)构造能暴露隐性错误的判别性测试,以及故障目标程序修复(FPR)在不同诊断条件下的修复表现。通过将具有挑战性的编程题与通过部分测试集但存在语义边界案例的LLM生成解配对,实现对诊断链断裂点的可控识别。对12个前沿大模型的评估显示,故障目标推理能力并不随通用编程能力提升而增长:模型几乎以天花板水平生成语法正确测试输入,但在判别性生成上急剧下降;故障假设生成而非输出验证成为主要瓶颈。测试引导修复揭示互补发现:当自生成测试成功触发故障时,修复效果可媲美或超越外部测试指导的修复;但未能触发故障的测试会显著恶化修复性能,低于无指导基线。结果重新定义了自主调试的核心挑战:真正瓶颈并非代码生成或测试有效性,而是故障目标推理能力,当前所有前沿模型均存在明显不足。

原文摘要 · Abstract (English)

As Large Language Models shift the programming toward human-guided ''vibe coding'', agentic coding tools increasingly rely on models to self-diagnose and repair their own subtle faults -- a capability central to autonomous software engineering yet never systematically evaluated. We present \name{}, the first empirical decomposition that jointly evaluates two coupled tasks: \emph{Fault-Triggering Test Generation (FT-Test)} constructing a discriminative witness that exposes a latent bug, and \emph{Fault-targeted Program Repair (FPR)}, repairing it under varying diagnostic conditions. \name{} pairs competitive programming problems with LLM-generated solutions that pass partial test suites but fail on semantic edge cases, enabling controlled identification of where the diagnostic chain breaks down. Evaluating 12 frontier LLMs, we find that fault-targeted reasoning does not scale with general coding ability. Models produce syntactically valid test inputs at near-ceiling rates yet collapse on discriminative generation, with fault hypothesis generation -- not output validation -- as the dominant bottleneck. Test-guided repair reveals a complementary insight: when self-generated tests successfully witness a fault, the resulting repair matches or outperforms repair guided by externally provided tests, but tests that fail to witness the fault actively degrade repair below unguided baselines. Together, these results reframe the challenge of autonomous debugging: the binding bottleneck is not code synthesis or test validity but fault-target reasoning, a capability that remains deficient across all frontier models. As Large Language Models shift the programming toward human-guided ''vibe coding'', agentic coding tools increasingly rely on models to self-diagnose and repair their own subtle faults -- a capability central to autonomous software engineering yet never systematically evaluated.

自主编程大模型调试评估故障推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。