评测大模型生成CUDA代码与专家水平的差距，发现仍需更强硬件推理能力。
CUDAHercules: Benchmarking Hardware-Aware Expert-level CUDA Optimization for LLMs
- 构建跨架构的CUDA优化基准，覆盖单核到完整应用
- 大模型生成代码常通过编译测试但性能远低于专家级实现
- 当前方法依赖工具反馈易退化为低效实现，需更智能的硬件理解
大型语言模型在自动化CUDA编程方面展现出潜力，但即使是最强大的编码模型（如Claude-Opus-4.6）仍难以达到专家级、架构感知的优化水平。我们提出CUDAHercules基准，用于评估生成的CUDA代码与端到端人类专家最先进系统之间的差距。该基准涵盖Ampere、Hopper和Blackwell GPU上的单核、模块级算子、完整应用及未解挑战任务，并通过领域特定语义验证器控制端到端任务。评估Claude-Opus-4.6和GPT-5.4发现：模型生成的代码虽能编译并通过测试，但极少复现专家所需的优化策略以匹配其性能。应用语义进一步降低成功率，迭代或工具辅助反馈虽提升正确性，却趋向于低效的降级实现。结果表明，自动化CUDA编程仍未完全解决，亟需更强的硬件推理能力、更好的工具使用方式，以及将代码理解与硬件架构驱动智能相连接的训练目标。
原文摘要 · Abstract (English)
Large language models show promise for automated CUDA programming, however even the strongest coding models (e.g., Claude-Opus-4.6) may still fall short of expert-level, architecture-aware optimization. We introduce CUDAHercules, a benchmark that evaluates generated CUDA against end-to-end human-expert SOTA systems. It spans single kernels, module-level operators, full applications, and unsolved challenge tasks across Ampere, Hopper, and Blackwell GPUs, with end-to-end tasks gated by domain-specific semantic validators. Evaluating models such as Claude-Opus-4.6 and GPT-5.4 shows a large gap between runnable CUDA and expert CUDA engineering: models often compile and pass tests, but rarely recover the optimization strategies needed to match expert performance. Application semantics further reduce success, and iterative or tool-augmented feedback can improve correctness while drifting toward slow fallback implementations. These results show that automated CUDA programming remains far from fully solved and requires stronger hardware reasoning, better tool use, and training objectives that connect code understanding to hardware architecture-grounded intelligence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。