测试智能体在真实硬件设计中的表现，发现软件经验难迁移，定位精度和反馈机制更关键。
Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench
- 构建了包含511个经验证Verilator实例的硬件工程评测基准Phoenix-bench。
- 同一智能体在硬件任务中成功率下降37%至58%，因信号流传播机制不同。
- 测试反馈比精准定位更能提升修复率，单轮反馈可提效42%至45%。
我们探究为软件工程设计的智能体是否适用于真实的硬件工程。现有硬件大模型评测仅关注子任务，缺乏对仓库导航、层次感知定位、电子设计自动化（EDA）可执行验证及维护式修补的联合考察。本文提出 extbf{Phoenix-bench}，一个由114个GitHub仓库中的511个经验证的Verilator实例组成的同步语料库，每个实例配备开发者补丁、设计流程标签、失败到通过及通过到失败的测试平台，以及基于Docker固定版本的EDA环境，确保解决率差异反映智能体行为而非工具链差异。利用该基准，我们对四个商业智能体和八个开源智能体结构在四种LLM基础上进行统一评估，并引入两项诊断干预：文件级黄金标准定位与一轮测试日志反馈。结果发现：(i) 软件与硬件是本质不同的工程任务：相同智能体在从SWE-bench Verified到Phoenix-bench时成功率下降37%至58%，因硬件错误通过信号流跨并行实例模块传播，而非软件调用图路径，且软件调优智能体仅停于症状文件而未追溯实例链；(ii) 失败集中于控制流/有限状态机（FSM）错误、验证测试平台错误，以及需跨层级信号流追踪与多文件协同编辑的难题；(iii) 定位粒度远比定位本身重要：完美文件级黄金标准仅带来+1.4%提升，因智能体常误改无需修改的文件，而一轮测试反馈使解决率提升42%至45%，因测试用例明确指出错误位置与修复目标。
原文摘要 · Abstract (English)
We ask whether agentic AI systems built for software engineering transfer to realistic hardware engineering. Existing hardware LLM benchmarks isolate sub-tasks but none jointly requires repository navigation, hierarchy-aware localization, Electronic Design Automation (EDA) executable verification, and maintenance-style patching. We introduce \textbf{Phoenix-bench}, a synchronized corpus of 511 verified Verilator instances from 114 GitHub repositories, each shipped with the developer patch, design-flow labels, fail-to-pass and pass-to-pass testbenches, and a Docker-pinned EDA environment so resolved-rate differences reflect agent behavior rather than toolchain availability. Using Phoenix-bench we run a uniform evaluation of four commercial agents and eight open-source agentic structures across four LLM backbones, plus two diagnostic interventions (file-level oracle localization and one round of testbench-log feedback). Three findings emerge. (i)~Software and hardware are fundamentally different engineering tasks: the same agent loses 37\% to 58\% from SWE-bench Verified to Phoenix-bench because hardware bugs propagate across parallel instantiated modules through signal flow rather than along a software-style call graph, and software-tuned agents stop at the symptom file instead of tracing back through the instantiation chain. (ii)~Failures concentrate on design control-flow / finite state machine (FSM) bugs, verification testbench bugs, and hard cases that demand cross-hierarchy signal-flow tracking and coordinated multi-file edits. (iii)~Localization granularity matters far more than localization itself: a perfect file-level oracle yields only $+1.4$\% because the agent then breaks files that did not need editing, while a single round of test case feedback lifts resolved rate by $42$\% to $45$\% because the test case tells \emph{where} the bug is and \emph{what} the fix has to look like.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。