分析705个开源大模型故障，发现部署问题比模型缺陷更关键。
Why Does the LLM Stop Computing: An Empirical Study of User-Reported Failures in Open-Source LLMs
- 通过实证研究定位部署栈系统性脆弱点。
- 发现运行崩溃多因基础设施摩擦，功能错误多因分词器缺陷。
- 适合关注模型落地可靠性的开发者和运维人员。
开源大语言模型的普及使用户可在本地部署与微调模型，但暴露于首个部署环节的不确定性中。与黑箱API使用不同，用户自管的编排可靠性成为关键盲区。我们首次对DeepSeek、Llama和Qwen生态中的705个真实故障进行大规模实证研究。分析揭示范式转变：白盒编排将可靠性瓶颈从模型算法缺陷转移至部署栈的系统性脆弱性。识别出三个关键现象：(1) 诊断异化：运行时崩溃显著标志基础设施摩擦，而功能错误则为内部分词器缺陷的特征。(2) 系统同质性：根本原因在不同系列中趋同，表明可靠性障碍源于共享生态而非特定架构。(3) 生命周期升级：障碍从微调阶段的配置困境，演变为推理阶段的复合环境不兼容。基于公开数据集，这些洞察为提升大模型部署可靠性提供可操作指导。
原文摘要 · Abstract (English)
The democratization of open-source Large Language Models (LLMs) allows users to fine-tune and deploy models on local infrastructure but exposes them to a First Mile deployment landscape. Unlike black-box API consumption, the reliability of user-managed orchestration remains a critical blind spot. To bridge this gap, we conduct the first large-scale empirical study of 705 real-world failures from the open-source DeepSeek, Llama, and Qwen ecosystems. Our analysis reveals a paradigm shift: white-box orchestration relocates the reliability bottleneck from model algorithmic defects to the systemic fragility of the deployment stack. We identify three key phenomena: (1) Diagnostic Divergence: runtime crashes distinctively signal infrastructure friction, whereas incorrect functionality serves as a signature for internal tokenizer defects. (2) Systemic Homogeneity: Root causes converge across divergent series, confirming reliability barriers are inherent to the shared ecosystem rather than specific architectures. (3) Lifecycle Escalation: Barriers escalate from intrinsic configuration struggles during fine-tuning to compounded environmental incompatibilities during inference. Supported by our publicly available dataset, these insights provide actionable guidance for enhancing the reliability of the LLM landscape.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。