首个面向物理模拟的低成本评估基准，测试大模型在真实预算下的表现。
SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs
- 构建12个物理仿真器的基准，量化大模型调参的计算与时间成本
- 单轮模式成功率45%-62%，高精度要求下降至34%-50%，多轮提升至66%-81%
- 开源工具包支持低成本智能体设计，适合科研与工程部署
评估大模型在科学任务中的表现长期聚焦于令牌成本，忽略了仿真时间与实验资源等工具使用成本。因此，在真实预算约束下，传统的通过率指标（pass@k）变得不切实际。为此，我们提出SimulCost，首个针对物理模拟中成本敏感参数调优的基准。SimulCost对比大模型与传统扫描方法在准确性和计算成本上的表现，涵盖来自流体动力学、固体力学和等离子体物理的11个仿真器，共2,643个单轮（初始猜测）和2,304个多轮（试错调整）任务，其成本为解析定义且平台无关。第十二个仿真器为仅可通过墙钟测量的生产级等离子体代码，单独报告。前沿大模型在单轮模式下成功率达45%-62%，在高精度要求下降至34%-50%，表明初始猜测不可靠，尤其对高精度任务；多轮模式成功率提升至66%-81%，但比传统扫描慢1.5-2.7倍，成本过高。我们还研究了参数组相关性以探索知识迁移潜力，以及上下文示例和推理努力的影响，为部署与微调提供实践指导。我们开源SimulCost作为静态基准与可扩展工具包，推动改进成本感知智能体设计及新仿真环境拓展。代码与数据见https://github.com/Rose-STL-Lab/SimulCost-Bench。
原文摘要 · Abstract (English)
Evaluating LLM agents for scientific tasks has focused on token costs while ignoring tool-use costs like simulation time and experimental resources. As a result, metrics like pass@k become impractical under realistic budget constraints. To address this gap, we introduce SimulCost, the first benchmark targeting cost-sensitive parameter tuning in physics simulations. SimulCost compares LLM tuning cost-sensitive parameters against traditional scanning approach in both accuracy and computational cost, spanning 2,643 single-round (initial guess) and 2,304 multi-round (adjustment by trial-and-error) tasks across 11 simulators from fluid dynamics, solid mechanics, and plasma physics, whose costs are analytically defined and platform-independent. A twelfth simulator, a production plasma code measurable only by wall clock, is reported separately. Frontier LLMs achieve 45-62% success rates in single-round mode, dropping to 34-50% under high accuracy requirements, rendering their initial guesses unreliable especially for high accuracy tasks. Multi-round mode improves rates to 66-81%, but LLMs are 1.5-2.7x slower than traditional scanning, making them uneconomical choices. We also investigate parameter group correlations for knowledge transfer potential, and the impact of in-context examples and reasoning effort, providing practical implications for deployment and fine-tuning. We open-source SimulCost as a static benchmark and extensible toolkit to facilitate research on improving cost-aware agentic designs for physics simulations, and for expanding new simulation environments. Code and data are available at https://github.com/Rose-STL-Lab/SimulCost-Bench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。