LLM生成硬件断言效果不佳,新模型可显著提升准确率。
Are LLMs Ready for Practical Adoption for Assertion Generation?
- 针对断言生成任务,定制化微调专用LLM模型
- 相比商用LLM,语法和语义正确率大幅提升
- 适合硬件验证工程师评估AI生成断言可靠性
断言是十余年来自行仿真与形式化验证硬件设计的默认手段。硬件验证质量——即对边界情况设计缺陷的检测与诊断——高度依赖断言质量。随着Transformer与大语言模型(LLMs)等生成式AI兴起,从设计源代码生成功能与安全断言的新方法受到关注。尽管已有研究使用商用现成(COTS)LLM进行断言生成,但尚无系统性研究量化其在生成语法与语义正确断言方面的有效性。本文首先回顾我们先前提出的AssertionBench,这是一个涵盖多种设计与断言的综合性基准,用于评估各类COTS LLM在从硬件源码生成断言任务中的表现。关键发现是:现有COTS LLM尚未具备实际部署条件,因生成大量语法或语义错误的断言。基于此洞察,我们提出AssertionLLM,首个专为断言生成任务定制的微调模型。初步实验结果表明,AssertionLLM在生成断言的语义与语法正确性上显著优于COTS LLM。
原文摘要 · Abstract (English)
Assertions have been the de facto collateral for simulation-based and formal verification of hardware designs for over a decade. The quality of hardware verification, i.e., detection and diagnosis of corner-case design bugs, is critically dependent on the quality of the assertions. With the onset of generative AI such as Transformers and Large-Language Models (LLMs), there has been a renewed interest in developing novel, effective, and scalable techniques of generating functional and security assertions from design source code. While there have been recent works that use commercial-of-the-shelf (COTS) LLMs for assertion generation, there is no comprehensive study in quantifying the effectiveness of LLMs in generating syntactically and semantically correct assertions. In this paper, we first discuss AssertionBench from our prior work, a comprehensive set of designs and assertions to quantify the goodness of a broad spectrum of COTS LLMs for the task of assertion generations from hardware design source code. Our key insight was that COTS LLMs are not yet ready for prime-time adoption for assertion generation as they generate a considerable fraction of syntactically and semantically incorrect assertions. Motivated by the insight, we propose AssertionLLM, a first of its kind LLM model, specifically fine-tuned for assertion generation. Our initial experimental results show that AssertionLLM considerably improves the semantic and syntactic correctness of the generated assertions over COTS LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。