评测大模型修复CUDA代码的真实能力,发现多数修复只是变慢而非真正修复。
CUDABeaver: Benchmarking LLM-Based Automated CUDA Debugging

- 构建真实失败的CUDA任务集,检验修复是否真正有效
- 发现40%修复因性能下降被拒,严格要求下成功率骤降
- 提出新评估协议,区分真修复与低效替代方案
CUDA程序调试长期困难,因硬件行为、编译决策、内存层次和异步执行间的微妙交互导致故障。随着GPU在科学计算、机器学习等领域的广泛应用,调试难度持续上升。现有大模型评估多忽略此场景:模型可通过退化修复通过测试,将原优化代码简化为更安全但更慢的版本。我们提出CUDABEAVER,一个基于真实失败工作区的CUDA调试基准,每个任务提供错误候选代码、原生构建/测试命令、原始错误证据及单个可编辑文件。该基准评估修复是否真正修复代码,或仅生成更慢的通过版本,结果按故障类型、调试轨迹、停滞模式和性能保留度报告。我们进一步提出pass@k(M,C,A)指标,明确修复器M、语料库C与协议轴A。在213个任务和7个前沿大模型上验证,协议感知评估揭示更真实的调试能力:高容忍性能损失时表现强劲,但微小性能约束即导致成功率大幅下降,分数最高波动达40个百分点。
原文摘要 · Abstract (English)
Debugging CUDA programs has long been challenging because failures often arise from subtle interactions among hardware behavior, compiler decisions, memory hierarchy, and asynchronous execution. More importantly, with the rapid expansion of GPU usage across scientific computing, machine learning, graphics, and systems workloads, CUDA debugging has become more challenging than ever. Current evaluations of LLM-based CUDA programming largely miss this setting: a model can pass correctness tests with repair by degeneration, simplifying the CUDA code into a safer but slower program that abandons the original optimization structure. We introduce CUDABEAVER, a benchmark for CUDA debugging from real failing workspaces produced during LLM-based CUDA generation. Each task provides the broken candidate, native build/test commands, raw error evidence, and a single editable file. CUDABEAVER evaluates whether a fixer truly repairs the failing CUDA code or merely finds a slower test-passing replacement, reporting results by failure category, debugging trajectory, stagnation mode, and performance preservation. We further propose pass@k(M,C,A), a protocol-conditional CUDA debugging metric by making the fixer M, corpus C, and protocol axes Aexplicit. Using this metric across 213 tasks and seven frontier LLMs, we show that protocol-aware evaluation gives a more faithful view of CUDA debugging ability: when performance-loss tolerance is high, fixers appear much stronger, but even a minor stricter performance requirement can sharply reduce measured success, shifting scores by up to 40 percentage points.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。