arXiv:2604.14709cs.AI2026-04被引 2

首个面向真实硬件漏洞修复的大型基准,测试大模型在复杂项目中的实战能力。

HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks

论文配图:HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks
图 1 · 摘自论文原文
  • 构建覆盖6个开源项目的容器化修复任务集,模拟真实开发流程。
  • 最优模型仅解决70.7%任务,复杂SoC项目成功率低于65%。
  • 失败源于定位、语义理解与多组件协同难题,为硬件智能提供改进方向。

现有硬件设计评估基准主要聚焦于孤立的模块级任务,如从规格生成HDL代码,缺乏对仓库级任务的覆盖。我们提出HWE-Bench,首个面向真实硬件漏洞修复的大型仓库级评估基准。该基准包含417个任务实例,源自六个主流开源项目的真实历史漏洞修复拉取请求,涵盖RISC-V核心、SoC及安全根信任模块,涉及Verilog/SystemVerilog和Chisel语言。每个任务均在全容器化环境中进行,要求代理解决真实缺陷报告,并通过项目原生仿真与回归测试验证正确性。基准通过高度自动化流水线构建,支持快速扩展至新仓库。我们评估了七种LLM搭配四种代理框架,发现最佳代理整体任务解决率为70.7%,在小型核心上超过90%,但在复杂SoC项目中低于65%。性能差异显著大于常见软件基准,困难程度由项目范围与漏洞类型分布决定,而非仅代码量。失败分析揭示问题集中在故障定位、硬件语义推理及跨RTL、配置与验证组件的协调三个阶段,为开发更懂硬件的智能体指明路径。

原文摘要 · Abstract (English)

Existing benchmarks for hardware design primarily evaluate Large Language Models (LLMs) on isolated, component-level tasks such as generating HDL modules from specifications, leaving repository-scale evaluation unaddressed. We introduce HWE-Bench, the first large-scale, repository-level benchmark for evaluating LLM agents on real-world hardware bug repair tasks. HWE-Bench comprises 417 task instances derived from real historical bug-fix pull requests across six major open-source projects spanning both Verilog/SystemVerilog and Chisel, covering RISC-V cores, SoCs, and security roots-of-trust. Each task is grounded in a fully containerized environment where the agent must resolve a real bug report, with correctness validated through the project's native simulation and regression flows. The benchmark is built through a largely automated pipeline that enables efficient expansion to new repositories. We evaluate seven LLMs with four agent frameworks and find that the best agent resolves 70.7% of tasks overall, with performance exceeding 90% on smaller cores but dropping below 65% on complex SoC-level projects. We observe larger performance gaps across models than commonly reported on software benchmarks, and difficulty is driven by project scope and bug-type distribution rather than code size alone. Our failure analysis traces agent failures to three stages of the debugging process: fault localization, hardware-semantic reasoning, and cross-artifact coordination across RTL, configuration, and verification components, providing concrete directions for developing more capable hardware-aware agents.

硬件智能大模型评估漏洞修复LLM代理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。