首次系统评估智能体对Verilog代码生成的影响,发现合理设计才能提升性能。
Exploring the Agentic Frontier of Verilog Code Generation
- 构建专用工具链的智能体框架,通过结构化提示提升生成质量。
- 前沿模型在智能体模式下性能反而下降,优化提示比盲目加工具更有效。
- 开源模型因崩溃率高、工具理解弱,与闭源模型差距明显,适合研究者参考。
大型语言模型在Python、C++等通用编程语言代码生成方面进展迅速,很大程度得益于结合领域工具的智能体框架。硬件设计语言如Verilog近年也取得进步,但智能体对这类任务的影响尚不明确。本文首次针对Verilog生成任务进行系统性评估,采用新提出的CVDP基准,并引入多个开源硬件设计智能体工具链,为未来研究提供模型无关基线。通过控制实验,我们分析了结构化提示与工具设计对性能的影响,考察了智能体失败模式与工具使用规律,对比了开源与闭源模型表现,并给出成功与失败案例。结果表明,对前沿模型简单套用智能体框架会降低性能(相比优化提示的直接推理),而结构化智能体框架可显著提升甚至超越非智能体基线。开源模型与闭源模型的差距主要源于更高崩溃率和较弱的工具输出解析能力。本研究为未来定制化Verilog生成智能体指明方向。
原文摘要 · Abstract (English)
Large language models (LLMs) have made rapid advancements in code generation for popular languages such as Python and C++. Many of these recent gains can be attributed to the use of ``agents'' that wrap domain-relevant tools alongside LLMs. Hardware design languages such as Verilog have also seen improved code generation in recent years, but the impact of agentic frameworks on Verilog code generation tasks remains unclear. In this work, we present the first systematic evaluation of agentic LLMs for Verilog generation, using the recently introduced CVDP benchmark. We also introduce several open-source hardware design agent harnesses, providing a model-agnostic baseline for future work. Through controlled experiments across frontier models, we study how structured prompting and tool design affect performance, analyze agent failure modes and tool usage patterns, compare open-source and closed-source models, and provide qualitative examples of successful and failed agent runs. Our results show that naive agentic wrapping around frontier models can degrade performance (relative to standard forward passes with optimized prompts), but that structured harnesses meaningfully match and in some cases exceed non-agentic baselines. We find that the performance gap between open and closed source models is driven by both higher crash rates and weaker tool output interpretation. Our exploration illuminates the path towards designing special-purpose agents for verilog generation in the future.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。