用真实工具链评测大模型生成可综合Verilog代码的能力。
Benchmarking LLMs for Verilog Design Flows
- 构建端到端验证流程，结合语法修复与仿真验证。
- 1.1B小模型语法正确率最高达80%，超越8B模型。
- 适合硬件设计与AI生成代码研究者参考。
大型语言模型（LLMs）在代码生成方面展现潜力，但其生成正确、可综合的硬件描述语言（HDL）代码能力仍缺乏系统评估。现有评测多依赖pass@k指标，缺少完整工具链验证。本文提出一个可复现的基准平台，针对50个精心设计的任务（包含组合逻辑、时序电路、有限状态机及混合设计），在三个不同规模的开源LLM（Llama-3-8B、StarCoder2-7B、TinyLlama-1.1B）上评估其生成Verilog RTL的能力。该流程包括约束提示、后处理及基于波形分析、形式等价验证和抽象语法树（AST）修复的语义感知迭代优化，通过Verilator编译与Icarus Verilog仿真验证生成代码。在12个基准任务共1610次运行中，语法有效性从0%提升至平均70.43%，仿真通过率提升至51.8%。值得注意的是，1.1B参数的TinyLlama在语法正确率上达到80.0%，功能正确性接近8B模型。平台与数据集均开源，支持硬件设计流程中生成式AI的可复现评估。
原文摘要 · Abstract (English)
Large language models (LLMs) show promise in code generation, but their capabilities to produce correct, synthesizable hardware description language (HDL) code still remain to be properly benchmarked. Existing evaluations are primarily relying on pass@k metrics and lack proper end-to-end toolchain validation. This paper presents a reproducible benchmarking platform that evaluates open-source LLMs on Verilog RTL generation across 50 curated tasks consisting of combinational, sequential, finite state machine (FSM), and mixed designs. The pipeline consisting of constrained prompting, post-processing, and semantic-aware iterative refinement with waveform analysis, formal equivalence verification, and Abstract Syntax Tree (AST)-based repair validates the generated code via Verilator compilation and Icarus Verilog simulation. Across the 12 benchmarks and the 1,610 total runs evaluating three models of different sizes (Llama-3-8B, StarCoder2-7B, and TinyLlama-1.1B), the pipeline improved syntax validity from 0% to a 70.43% average and simulation pass rate to 51.8% across three open-source models. Most notably TinyLlama (1.1B parameters) achieved the highest individual syntax validity at 80.0%, with functional correctness comparable to the 8B model. The platform and dataset are open-source, enabling reproducible evaluation of generative AI for hardware design workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。