测试大模型对硬件性能的推理能力，发现其思路合理但建模稳定性差。
PerfReasoning: How Well Do LLMs Reason on Hardware Performance?
- 构建新基准评估大模型直接推理性能或生成性能代码
- 闭源模型推理准确率超90%，但建模成功率普遍低于15%
- 微调和强化学习能提升建模效果，盲目自修正无效
性能建模是硬件设计与软件优化的核心，需对计算、数据重用、存储及数据移动进行结构化推理。我们提出PerfReasoning基准，评估大模型作为性能推理者或性能模型代码生成器的能力。在给定工作负载、架构与映射规格下，模型需比较不同映射方案并预测外部流量与缓冲区需求。最强闭源模型在推理问答任务中超过90%准确率，最佳开源模型达82.4%。但模型构建难度显著更高：尽管GPT-5.6 Sol通过率超80%，其余配置平均低于15%且结果波动剧烈。特定任务强化学习使4B模型的映射推理准确率提升15.7个百分点，而无反馈的多轮自修正提示则不可靠。PerfReasoning揭示了模型在合理架构推理与可靠性能建模之间的差距。我们将公开发布该基准，以支持可复现评估并追踪未来进展。
原文摘要 · Abstract (English)
Performance modeling is central to hardware design and software optimization, yet constructing these models requires structured reasoning about computation, data reuse, storage, and movement. We introduce PerfReasoning, a benchmark that evaluates LLMs both as direct performance reasoners and as generators of analytical performance-model code. Given workload, architecture, and mapping specifications, models compare mappings and predict off-chip traffic and buffer requirements. The strongest closed-source models exceed 90% on reasoning-based Q&A, and the best open-weight model reaches 82.4%. However, model construction is substantially harder: while GPT-5.6 Sol exceeds 80% pass rate, all other model configurations average below 15% and vary markedly across runs. Task-specific RL raises a 4B model's mapping-reasoning accuracy by 15.7 points, whereas feedback-free multi-round self-revision prompting is not reliably effective. PerfReasoning exposes the gap between plausible architectural reasoning and reliable performance-model construction. We will publicly release the benchmark to support reproducible evaluation and track future progress.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。