对比云与本地大模型在系统动力学辅助中的表现，发现本地模型可媲美中等云服务。
Benchmarking System Dynamics AI Assistants: Cloud Versus Local LLMs on CLD Extraction and Discussion
- 构建两个专用基准测试，评估模型提取因果回路图和交互讨论能力。
- 本地模型在因果图提取上达77%，讨论中建模步骤正确率50%-100%，但纠错能力弱于云模型。
- 揭示后端框架比量化级别对性能影响更大，提供苹果芯片部署指南。
我们系统评估了主流大语言模型家族——包括专有云API与本地开源模型——在两个专为系统动力学AI辅助设计的基准上的表现： extbf{CLD Leaderboard}（53项测试，结构化因果回路图提取）与 extbf{Discussion Leaderboard}（互动模型讨论、反馈解释与建模指导）。在因果图提取任务中，云模型整体通过率为77%–89%；最佳本地模型（Kimi K2.5 GGUF Q3，零样本引擎）达到77%，与中等云服务相当。在讨论任务中，最佳本地模型在建模步骤上正确率达50%–100%，反馈解释准确率47%–75%，但错误修复仅0%–50%，该类任务受长上下文提示影响显著，暴露出本地部署的内存瓶颈。本文核心贡献在于系统分析模型类型对性能的影响：比较推理型与指令微调架构、GGUF（llama.cpp）与MLX（mlx_lm）后端、不同量化等级（Q3/Q4_K_M/MLX-3bit/MLX-4bit/MLX-6bit）在同一模型族下的表现。结果表明后端选择的实际影响大于量化级别：mlx_lm不强制JSON格式约束，需显式提示；而llama.cpp语法约束采样虽可靠生成JSON，但在密集模型的长上下文提示下可能导致无限生成。本文完整记录所有本地模型的参数扫描（t, p, k），清洗后的耗时数据（剔除卡死请求），并提供在Apple Silicon上运行671B–123B参数模型的实践指南。
原文摘要 · Abstract (English)
We present a systematic evaluation of large language model families -- spanning both proprietary cloud APIs and locally-hosted open-source models -- on two purpose-built benchmarks for System Dynamics AI assistance: the \textbf{CLD Leaderboard} (53 tests, structured causal loop diagram extraction) and the \textbf{Discussion Leaderboard} (interactive model discussion, feedback explanation, and model building coaching). On CLD extraction, cloud models achieve 77--89\% overall pass rates; the best local model reaches 77\% (Kimi~K2.5~GGUF~Q3, zero-shot engine), matching mid-tier cloud performance. On Discussion, the best local models achieve 50--100\% on model building steps and 47--75\% on feedback explanation, but only 0--50\% on error fixing -- a category dominated by long-context prompts that expose memory limits in local deployments. A central contribution of this paper is a systematic analysis of \textit{model type effects} on performance: we compare reasoning vs.\ instruction-tuned architectures, GGUF (llama.cpp) vs.\ MLX (mlx\_lm) backends, and quantization levels (Q3 / Q4\_K\_M / MLX-3bit / MLX-4bit / MLX-6bit) across the same underlying model families. We find that backend choice has larger practical impact than quantization level: mlx\_lm does not enforce JSON schema constraints, requiring explicit prompt-level JSON instructions, while llama.cpp grammar-constrained sampling handles JSON reliably but causes indefinite generation on long-context prompts for dense models. We document the full parameter sweep ($t$, $p$, $k$) for all local models, cleaned timing data (stuck requests excluded), and a practitioner guide for running 671B--123B parameter models on Apple~Silicon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。