提出Pufibara框架，提升物理系统建模代理的准确性与效率
Beyond Executable Models: The Pufibara Agent Harness and the Modelica Agent Workflow Benchmark for Physical System Modeling
- 构建持久工程状态的代理框架，确保每轮迭代可追溯
- 在232项任务中，比Claude Code多通过17项，资源消耗低76%-82%
- 适合需要高物理一致性的仿真建模场景，如工业设计与系统工程
AI代理在仿真驱动的工程中日益重要。物理系统建模不同于通用代码生成，其正确性不仅依赖语法和可执行性，还要求物理一致性与情境相关行为。我们以Modelica为例，发现模型虽能编译运行，仍可能违背物理或工程要求。随着版本迭代，代理易丢失需求或依赖过时仿真结果。为此，我们提出Pufibara代理框架，实现跨版本持久化工程状态，为每次执行和仿真证据绑定对应候选，使提交成为显式动作。为评估端到端建模流程，我们提出基于源码的现实任务构建方法，构建了包含232个任务的Modelica代理工作流基准（Modelica Agent Workflow Benchmark），涵盖模型修复、生成与调优。每个提交候选由基准独立评测器评分。在两个匹配的大型语言模型后端下，使用DeepSeek v4 Flash时，Pufibara通过202项任务，优于Claude Code的185项；使用Claude Sonnet 5时，分别通过202项与187项。在仓库报告的令牌统计下，Pufibara逻辑令牌总量降低76.4%-82.5%，序列运行时间减少6.1%-58.4%。结果表明，即使在相同模型后端下，完整代理框架在任务成功率与资源消耗上仍有显著差异。
原文摘要 · Abstract (English)
AI agents are increasingly used for simulation-driven engineering. Physical system modeling presents different requirements from general-purpose code generation in software engineering, because correctness depends not only on syntax and executability but also on physical consistency and scenario-dependent behavior. We study this challenge in Modelica, an equation-based modeling language in which a model may compile and simulate while still violating its intended physics or engineering requirements. Across successive revisions, an agent may lose track of requirements or rely on simulation evidence produced by an outdated candidate. To address this challenge, we present Pufibara, an agent harness that maintains persistent engineering state across revisions, associates execution and simulation evidence with the candidate that produced it, and makes submission an explicit agent action. To evaluate end-to-end Modelica agent workflows, we also propose a source-grounded method for constructing realistic and independently evaluable tasks. We use this method to build the 232-task Modelica Agent Workflow Benchmark, spanning Model Repair, Model Generation, and Model Tuning. Each submitted candidate is scored by a benchmark-owned evaluator outside the agent loop. We compare Pufibara with Claude Code as complete harnesses under two matched large language model (LLM) backends. With DeepSeek v4 Flash, Pufibara passes 202 tasks, compared with 185 for Claude Code. With Claude Sonnet 5, Pufibara passes 202 tasks, compared with 187 for Claude Code. Under the repository-reported token accounting, Pufibara records 76.4%-82.5% lower logical-token totals. Its sequential runtime is 6.1%-58.4% lower. These findings show that, even under matched LLM backends, complete agent harnesses can differ substantially in both task success and resource use for physical system modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。