LLM生成的GPU代码常因测试漏洞被误判为正确，实际存在可复现的错误。
The Correctness Illusion in LLM-Generated GPU Kernels
- 用可控数据集和精准参考实现对代码正确性的严格验证
- 在5种GPU上均发现9个伪正确错误，100%检出率且无误报
- 适合关注AI生成代码可靠性与测试方法的研究者
现有基准（KernelBench、TritonBench、GEAK）通过固定形状、小样本的allclose式检查评估大模型生成的GPU内核正确性，其参数（形状、数据类型、容差）对每个内核固定。本文构建包含24个Triton和CPU替代内核的受控语料库（15个正确控制样本，9个含已知转录错误的错误变体），采用操作模式感知的种子模糊测试，以高精度（fp64）CPU参考和每（操作，数据类型）绝对容差重新评估。该验证方式成功识别全部9个错误变体，15个正确控制样本全通过，控制样本零误报。将语料扩展至26个操作（新增flash-attention对），在5类GPU（RTX 3060、A10、L40S、A100 SXM4、H100 NVL）上重复测试，结果一致：10个虚假正确被击破，16个控制样本无误报。该结果反映的是大模型风格转录错误被单一形状检测误判为正确的问题，而非特定部署模型的错误率。所有失败均可按种子逐字复现。
原文摘要 · Abstract (English)
Benchmarks for LLM-generated GPU kernels (KernelBench, TritonBench, GEAK) score correctness through fixed-shape, small-sample allclose-style checks. The number of inputs varies between benchmarks. The shape, dtype, and tolerance are fixed for each kernel. We test that oracle empirically. We construct a controlled corpus of 24 Triton and CPU stand-in kernels (15 correct controls and 9 LLM-style buggy variants seeded with documented transcription errors) and re-evaluate it under op-schema-aware seeded fuzzing with a high-precision (fp64) CPU reference and per-(op, dtype) absolute tolerances. The seeded oracle flags 9 of 9 buggy kernels and passes 15 of 15 correct controls, at zero precision cost on controls. We extend the corpus to 26 ops (adding a flash-attention pair) and re-run the same protocol on five GPU classes (RTX 3060, A10, L40S, A100 SXM4, H100 NVL). The verdicts are identical across all five GPUs: 10 of 10 illusions caught and 16 of 16 controls clean. The corpus result is about LLM-style transcription bugs that the allclose-on-one-shape oracle certifies as correct, not about the bug rate of any specific deployed LLM. Every flagged failure replays byte-for-byte from a stored seed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。