arXiv:2506.12320cs.SEcs.AI2025-06被引 7

首次系统分析大模型库漏洞,发现接口误用是主因。

The Foundation Cracks: A Comprehensive Study on Bugs and Testing Practices in LLM Libraries

  • 分析313个修复提交,归纳出5类症状、14类根因。
  • 接口误用占比超三成,测试用例不足导致41.73%漏洞未被发现。
  • 适合关注AI基础设施可靠性的开发者与质量保障团队。

大型语言模型(LLM)库已成为当前人工智能革命的基础设施,支撑模型部署、推理优化、微调及生产服务。尽管其至关重要,但频繁的质量问题和漏洞威胁着基于它们构建的AI系统的可靠性。本文首次对现代LLM库中的漏洞特征与测试实践进行全面实证研究,分析了两个广泛使用的库——HuggingFace Transformers与vLLM中提取的313个修复提交。通过严谨的手动分析,建立了涵盖5类症状与14类根因的分类体系。主要发现显示,接口误用成为最主要的根因(占比32.17%-48.19%),标志着从传统深度学习框架以算法缺陷为主,转向以接口问题为核心。同时,我们检查了7,748个测试函数,识别出7种测试断言类型,其中预定义预期输出(如特定张量和文本字符串)最为常见。评估表明,多数漏洞未被检测到,原因在于测试用例不足(41.73%)、缺乏测试驱动(32.37%)及弱断言(25.90%)。基于此,本文提出提升LLM库质量保障的建议。

原文摘要 · Abstract (English)

Large Language Model (LLM) libraries have emerged as the foundational infrastructure powering today's AI revolution, serving as the backbone for LLM deployment, inference optimization, fine-tuning, and production serving across diverse applications. Despite their critical role in the LLM ecosystem, these libraries face frequent quality issues and bugs that threaten the reliability of AI systems built upon them. To address this knowledge gap, we present the first comprehensive empirical investigation into bug characteristics and testing practices in modern LLM libraries. We examine 313 bug-fixing commits extracted across two widely-adopted LLM libraries: HuggingFace Transformers and vLLM.Through rigorous manual analysis, we establish comprehensive taxonomies categorizing bug symptoms into 5 types and root causes into 14 distinct categories.Our primary discovery shows that API misuse has emerged as the predominant root cause (32.17%-48.19%), representing a notable transition from algorithm-focused defects in conventional deep learning frameworks toward interface-oriented problems. Additionally, we examine 7,748 test functions to identify 7 distinct test oracle categories employed in current testing approaches, with predefined expected outputs (such as specific tensors and text strings) being the most common strategy. Our assessment of existing testing effectiveness demonstrates that the majority of bugs escape detection due to inadequate test cases (41.73%), lack of test drivers (32.37%), and weak test oracles (25.90%). Drawing from these findings, we offer some recommendations for enhancing LLM library quality assurance.

LLM库漏洞分析测试实践

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。