用新指标评估AI生成系统动力学模型的准确性与指令遵循能力。
How Well Can AI Build SD Models?
- 提出因果转换与指令遵循双指标评估AI建模质量。
- GPT-4.5-preview表现最佳，整体准确率达92.9%。
- 适合关注AI辅助建模可靠性的研究者与实践者。
随着系统动力学（SD）自动化推进，人工智能虽提升效率，但可能因数据缺失或模型缺陷引入偏差。忽略多视角与数据的模型会威胁建模质量，无论由人或AI构建。为评估AI构建SD模型的能力，本文提出两项评价指标：技术正确性（因果转换）与指令遵循度（符合性）。我们开发了开源项目sd-ai，推动SD社区协作，利用ChatGPT等AI工具进行动态建模。同时建立评估理论并设计全面测试套件，用于评估sd-ai生态中各类工具。测试11个LLM在因果转换和指令遵循上的表现：gpt-4.5-preview总体得分92.9%，表现最优；o1在因果转换上达100%；gpt-4o识别出所有因果链，但在减少项的正向极性判断上存在困难。尽管gpt-4.5-preview和o1最准确，但gpt-4o成本最低。结果揭示不同LLM间显著差异，强调持续评估对负责任地发展AI建模工具至关重要。为此，我们发起跨开发者、建模者与利益相关方的开放协作，旨在标准化评估体系，以提升AI工具对建模过程的贡献。
原文摘要 · Abstract (English)
Introduction: As system dynamics (SD) embraces automation, AI offers efficiency but risks bias from missing data and flawed models. Models that omit multiple perspectives and data threaten model quality, whether created by humans or with the assistance of AI. To reduce uncertainty about how well AI can build SD models, we introduce two metrics for evaluation of AI-generated causal maps: technical correctness (causal translation) and adherence to instructions (conformance). Approach: We developed an open source project called sd-ai to provide a basis for collaboration in the SD community, aiming to fully harness the potential of AI based tools like ChatGPT for dynamic modeling. Additionally, we created an evaluation theory along with a comprehensive suite of tests designed to evaluate any such tools developed within the sd-ai ecosystem. Results: We tested 11 different LLMs on their ability to do causal translation as well as conform to user instruction. gpt-4.5-preview was the top performer, scoring 92.9% overall, excelling in both tasks. o1 scored 100% in causal translation. gpt-4o identified all causal links but struggled with positive polarity in decreasing terms. While gpt-4.5-preview and o1 are most accurate, gpt-4o is the cheapest. Discussion: Causal translation and conformance tests applied to the sd-ai engine reveal significant variations across lLLMs, underscoring the need for continued evaluation to ensure responsible development of AI tools for dynamic modeling. To address this, an open collaboration among tool developers, modelers, and stakeholders is launched to standardize measures for evaluating the capacity of AI tools to improve the modeling process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。