评测大模型生成的硬件代码效率，填补了性能评估空白。
Pluto: A Benchmark for Evaluating Efficiency of LLM-generated Hardware Code
- 构建114个带自检测试用例的硬件问题集，支持多目标优化评估。
- 大模型生成代码功能正确率78.3%，但面积、延迟、功耗效率仅63.8%~65.9%。
- 适合关注硬件自动化设计的大模型研究者与芯片工程师使用。
大型语言模型（LLMs）正被用于自动化硬件设计任务，包括Verilog代码生成。尽管早期基准主要关注功能正确性，高效硬件设计还需在面积、延迟和功耗等综合指标上优化。现有基准在全面评估这些方面存在不足：常缺乏优化基线或验证用测试用例。为此，我们提出Pluto，一个评估大模型生成硬件代码效率的基准与评估框架。Pluto包含114个问题的综合性评估集，配有自检测试用例及多个帕累托最优参考实现。实验结果表明，当前先进大模型的功能正确率可达78.3%（pass@1），但其合成效率仍落后于人工专家设计，面积效率为63.8%、延迟效率为65.9%、功耗效率为64.0%（eff@1）。这凸显了如Pluto般注重效率的评估框架对推动面向硬件的大模型研究的重要性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly used to automate hardware design tasks, including the generation of Verilog code. While early benchmarks focus primarily on functional correctness, efficient hardware design demands additional optimization for synthesis metrics such as area, delay, and power. Existing benchmarks fall short in evaluating these aspects comprehensively: they often lack optimized baselines or testbenches for verification. To address these gaps, we present Pluto, a benchmark and evaluation framework designed to assess the efficiency of LLM-generated Verilog designs. Pluto presents a comprehensive evaluation set of 114 problems with self-checking testbenches and multiple Pareto-optimal reference implementations. Experimental results show that state-of-the-art LLMs can achieve high functional correctness, reaching 78.3\% at pass@1, but their synthesis efficiency still lags behind expert-crafted implementations, with area efficiency of 63.8\%, delay efficiency of 65.9\%, and power efficiency of 64.0\% at eff@1. This highlights the need for efficiency-aware evaluation frameworks such as Pluto to drive progress in hardware-focused LLM research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。