arXiv:2606.08976cs.AI2026-06被引 1

构建超大规模硬件设计基准,挑战大模型在复杂电路生成与推理中的能力。

RTL-BenchLS: A Large-Scale Benchmark for RTL Reasoning and Generation with Large Language Models

论文配图:RTL-BenchLS: A Large-Scale Benchmark for RTL Reasoning and Generation with Large Language Models
图 1 · 摘自论文原文
  • 基于自监督任务扩展数据规模,突破传统标注瓶颈。
  • 覆盖1万+经形式化验证的Verilog设计,任务难度显著提升。
  • 适合研究芯片自动化设计的大模型开发者参考。

基于大语言模型的RTL生成与推理是硬件设计自动化的有前景方向。高质量基准对追踪进展至关重要,但现有基准在规模和任务范围上存在根本局限:设计普遍小而简单,任务集中于规格说明书到RTL的生成。前沿模型在现有基准上性能已饱和。由于需要对齐标签(如规格说明与测试平台),扩大基准规模极为困难,真实设计中此类高质量数据罕见。我们提出RTL-BenchLS,一个大规模基准,同时解决上述问题。它包含超过10,000个经过形式化等价性验证的Verilog设计,涵盖更大更复杂的电路。除原有生成任务外,新增三项联合评估推理与生成的新任务:往返推理、掩码内容推理和代码库问题修复。前两项为自监督任务,直接缓解扩展瓶颈。所有任务均通过形式化等价检查验证,无需人工测试平台。我们在8个大模型上评估,即使最佳模型在自然语言往返推理上仅达23%,掩码内容推理28%,代码库问题修复仅12%。该基准远超现有水平,为未来方法发展提供空间与指引。

原文摘要 · Abstract (English)

LLM-based RTL generation and reasoning is a promising direction for hardware design automation. High-quality benchmarks are critical infrastructure for tracking progress in this direction. However, existing RTL benchmarks face inherent limitations in both scale and task scope. The designs they cover are typically small and simple, and the tasks focus almost entirely on specification-to-RTL generation. Frontier models' performance already saturates on the existing benchmarks. Scaling these benchmarks up is fundamentally difficult because aligned labels are required for benchmarking, such as specifications and testbenches. Such aligned high-quality data are rarely available for real-world designs. We introduce RTL-BenchLS, a large-scale benchmark addressing both limitations above. It contains over 10,000 formally verified Verilog designs, covering substantially larger and more complex designs than existing benchmarks. Beyond specification-to-RTL generation, we propose three novel tasks that jointly evaluate reasoning and generation: round-trip reasoning, masked-content reasoning, and repository-issue reasoning. The first two are self-supervised, which directly resolves the scaling bottleneck. All tasks are verified through formal equivalence checking without any manual testbenches. We evaluate eight LLMs on RTL-BenchLS. Even the best model reaches only 23% on natural-language round-trip reasoning, 28% on masked-content reasoning, and 12% on repository-issue fixing. RTL-BenchLS is substantially more challenging than existing benchmarks. It leaves ample room for future improvement and offers guidance for developing LLM-based methods for hardware design.

硬件自动化大模型基准测试Verilog

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。