测试大模型能否不运行代码就预测CUDA核的浮点运算量，发现其在复杂情况仍严重出错。
Counting Without Running: Evaluating LLMs' Reasoning About Code Complexity
- 设计gpuFLOPBench基准，让模型预测577个CUDA核的单双精度浮点操作数
- 最新模型对简单核预测准确，但遇到除法或数学函数时误差达数量级级
- 适合关注代码性能推理能力的开发者和工具研究者
现代GPU软件栈要求开发者在启动内核前就能预判性能瓶颈；上游误判浮点负载可能导致调优、调度甚至硬件采购失败。尽管代码生成技术快速进步，当前大语言模型很少被测试此类前瞻性推理能力。本文提出gpuFLOPBench基准，要求模型‘无需运行’即可预测来自HeCBench的577个CUDA核的单精度与双精度浮点运算次数（FLOP），并附带真实性能数据及八项执行属性，用以区分可直接分析的代码与依赖隐藏编译器或运行时行为的复杂核。评估现有闭源推理模型显示：最新模型在简单核上表现完美，但在涉及除法、数学函数或公共子表达式等隐含计算时仍出现多个数量级误差。结果揭示了现有代码助手的核心局限——无法内化硬件特定微码影响，并将gpuFLOPBench定位为开发具备专业开发者水准性能推理能力的LLM工具的专用测试平台。代码仓库见：https://github.com/Scientific-Computing-Lab/gpuFLOPBench
原文摘要 · Abstract (English)
Modern GPU software stacks demand developers who can anticipate performance bottlenecks before ever launching a kernel; misjudging floating-point workloads upstream can derail tuning, scheduling, and even hardware procurement. Yet despite rapid progress in code generation, today's Large Language Models (LLMs) are rarely tested on this kind of forward-looking reasoning. We close that gap with gpuFLOPBench, a benchmark that asks models to "count without running" by predicting single and double-precision FLOP counts for 577 CUDA kernels drawn from HeCBench, annotated with ground-truth profiles and eight execution attributes that distinguish trivially analyzable code from kernels whose FLOPs depend on hidden compiler or runtime behavior. Evaluating current closed-source reasoning models shows clear but uneven progress: the newest LLMs achieve perfect classification on straightforward kernels but still incur multiple order-of-magnitude errors whenever implicit FLOPs arise from division, intrinsic math functions, or common subexpressions. These results surface a core limitation of existing code assistants -- the inability to internalize hardware-specific microcode effects -- and position gpuFLOPBench as a focused testbed for developing LLM tooling that can reason about performance with the same rigor as experienced GPU developers. Sources are available at our repository: https://github.com/Scientific-Computing-Lab/gpuFLOPBench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。