新基准测试揭示大模型在芯片设计中真实表现差距
ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design
- 构建覆盖验证、调试、建模的三类工业级任务
- 顶级模型在代码生成上仅达30.74%，远低于旧基准95%以上
- 提供自动化数据生成工具，助力模型训练研究
尽管大型语言模型（LLMs）在硬件工程中展现巨大潜力，但现有基准已趋于饱和且任务类型有限，无法反映其在真实工业流程中的表现。为此，我们提出一个全面的AI辅助芯片设计基准，严格评估LLMs在Verilog生成、调试和参考模型生成三个关键任务上的能力。该基准包含44个具有复杂层次结构的真实模块、89个系统性调试案例，以及跨Python、SystemC和CXXRTL的132个参考模型样本。评估结果揭示显著性能差距：最先进模型Claude-4.5-opus在Verilog生成任务上仅达30.74%，在Python参考模型生成上仅为13.33%，远低于旧基准中超过95%的通过率。此外，为提升参考模型生成能力，我们还提供一套自动化高质量训练数据生成工具，推动该未充分探索领域的研究进展。代码已开源。
原文摘要 · Abstract (English)
While Large Language Models (LLMs) show significant potential in hardware engineering, current benchmarks suffer from saturation and limited task diversity, failing to reflect LLMs' performance in real industrial workflows. To address this gap, we propose a comprehensive benchmark for AI-aided chip design that rigorously evaluates LLMs across three critical tasks: Verilog generation, debugging, and reference model generation. Our benchmark features 44 realistic modules with complex hierarchical structures, 89 systematic debugging cases, and 132 reference model samples across Python, SystemC, and CXXRTL. Evaluation results reveal substantial performance gaps, with state-of-the-art Claude-4.5-opus achieving only 30.74\% on Verilog generation and 13.33\% on Python reference model generation, demonstrating significant challenges compared to existing saturated benchmarks where SOTA models achieve over 95\% pass rates. Additionally, to help enhance LLM reference model generation, we provide an automated toolbox for high-quality training data generation, facilitating future research in this underexplored domain. Our code is available at https://github.com/zhongkaiyu/ChipBench.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。