本地大模型可可靠完成硬件设计工具调用任务,但配置影响成功率。
Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling

- 构建MCP服务器模拟真实硬件工具状态与依赖逻辑
- 强模型在复杂流程中达近95%预期调用覆盖率
- 多代理分解提升弱模型表现,但增加调用开销
我们研究本地部署的大语言模型驱动的AI代理,能否在工业级工具调用场景中可靠自动化专家定义的硬件设计流程。工程师通过专用工具执行重复性、依赖有序的操作,如创建组件、添加引脚和布线。由于组件规格和命名规范涉及保密,常无法使用托管的专有API,因此采用本地部署模型。为此,我们构建了模型上下文协议(MCP)服务器,复现嵌入式系统开发中使用的专有硬件设计工具的状态与依赖逻辑,并构建涵盖单步编辑、多步依赖链、无效请求、拼写错误提示及多服务器工具环境的基准测试。评估七种开源模型,对比系统提示、工具描述细节、上下文范围及单代理与多代理架构等管道选择。结果表明,强模型可在基准工作流中实现接近完全的预期调用覆盖,但可靠性强烈依赖任务结构与代理配置。全面的工具描述能持续降低失败率,少样本提示对部分模型引发严重不作为,累积上下文会损害受限模型表现,而多代理分解有助于弱模型或长会话,代价是额外调用开销。这些发现为在有状态硬件设计环境中部署本地LLM代理提供了实用指导。
原文摘要 · Abstract (English)
We ask whether AI agents powered by locally deployed large language models can reliably automate expert-defined hardware design workflows in an industry-realistic tool-calling setting. In these environments, engineers issue repetitive, dependency-ordered operations---such as creating components, adding ports, and wiring connections---through specialised tools. Confidentiality constraints on component specifications and naming conventions often preclude hosted proprietary APIs, motivating the use of locally deployed models. To study this setting, we build a Model Context Protocol (MCP) server that reproduces the state and dependency logic of a proprietary hardware design tool used in embedded system development and construct a benchmark covering single-operation edits, multi-step dependency chains, invalid requests, misspelled prompts, and multi-server tool contexts. We evaluate seven open-source models comparing pipeline choices including system prompts, tool-description detail, context scope, and single-agent versus multi-agent architectures. Results show that strong models can achieve near-complete expected-call coverage on the benchmarked workflows, but reliability depends strongly on both task structure and agent configuration. Comprehensive tool descriptions consistently reduce failures, few-shot prompting can cause severe inaction for some models, cumulative context harms constrained models, and multi-agent decomposition helps weak workers or long sessions at the cost of additional calls. These findings provide practical guidance for deploying local LLM agents in stateful hardware design environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。