LLM在硬件编程中失败的根本原因被系统揭示，发现其能力受限于预训练知识而非对齐优化。
How LLMs Fail and Generalize in RTL Coding for Hardware Design?
- 提出基于可解性的问题分类法，区分语法、语义及可解/不可解功能错误
- 前沿模型在VerilogEval上初始通过率仅达90.8%，且无法突破
- 优化反而加剧深层功能错误，说明当前对齐策略治标不治本
将顺序编程先验转化为硬件设计的并行时序逻辑仍是大语言模型（LLM）的关键瓶颈。为探究此问题，我们引入一种基于可解性的新错误分类体系，受认知理论启发。该分类将失败归为语法、语义、可解功能和不可解功能四类。评估显示，在VerilogEval基准上存在严格的实证上限，前沿模型初始通过率稳定在90.8%。这些瓶颈由不可解功能错误构成，暴露了对测试时计算扩展免疫的持续知识缺口。此外，我们发现显著的表面收敛差距：优化可轻易消除语法错误，但同时加剧更深层的功能性缺陷。研究结果表明，对齐技术仅教会模型编译代码。尽管重复采样可修复可解错误，但寄存器传输级（RTL）编码能力仍严格受限于预训练知识。解决当前基于LLM的硬件生成流程挑战，需更多关注模型推理能力，而非对齐干预。
原文摘要 · Abstract (English)
Translating sequential programming priors into the parallel temporal logic of hardware design remains a crucial bottleneck for large language models(LLM). To investigate this, we introduce a new error taxonomy grounded in problem solvability, inspired by cognitive theory. Our taxonomy categorizes failures into syntactic, semantic, solvable functional, and unsolvable functional types. Evaluations reveal a strict empirical ceiling on the VerilogEval benchmark, as frontier models plateau at a 90.8% initial pass rate. These plateaus are defined by unsolvable functional errors, exposing persistent knowledge gaps immune to test time compute scaling. Furthermore, we expose a striking surface convergence gap: optimization readily eliminates syntax errors but concurrently exacerbates deeper functional failures. Our findings demonstrate that alignment techniques merely teach models to compile. While repeated sampling strategies can patch solvable errors, register-transfer level(RTL) coding capacity remains strictly bounded by pretraining knowledge. Addressing challenges in the current LLM based hardware generation pipeline requires more studies in model reasoning rather than alignment interventions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。