为大模型更新设计部署方治理框架,防范无声演进带来的功能退化。
Test Before You Deploy: Governing Updates in the LLM Supply Chain

- 定义模型行为规范(生产合约)与风险分类测试套件
- 通过兼容性关卡阻止不达标更新,实测可发现全局指标忽略的性能下降
- 适合关注模型部署安全与稳定性的研发团队参考
大语言模型作为软件系统的核心依赖,其托管服务在无明确版本变更的情况下持续更新,可能引发行为漂移,导致功能、格式或安全约束等方面的回归问题。现有方法多聚焦于回归测试或版本管理,缺乏部署方对模型透明演化过程中的兼容性管控机制。本文提出一种基于三要素的部署端治理框架:明确模型行为规则(生产合约)、按部署风险类别组织的定向测试(风险分类测试套件)、以及未达安全与性能标准则阻断更新的发布检查点(兼容性关卡)。通过多版本大模型的探索性验证,表明针对特定风险领域的定向测试能捕捉到整体指标遗漏的性能退化。同时识别出若干开放挑战:如何系统构建有效测试集、如何在非确定性系统中设定可靠性能阈值、以及在提供商透明度有限时如何检测与解释模型漂移。总体上,将大模型更新管理视为软件供应链治理问题,并提出可落地的部署方兼容性控制研究议程。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly used as core dependencies in software systems. However, the hosted LLM services evolve continuously through provider-side updates without explicit version changes. These silent updates can introduce behavioral drift, causing regressions in functionality, formatting, safety constraints, or other application-specific requirements. Existing approaches focus primarily on regression testing or versioning but do not provide deployer-side mechanisms for governing compatibility during opaque model evolution. This paper proposes a deployment-side governance framework based on three components: clearly defined rules for how the model is allowed to behave (production contracts), focused testing organized by deployment risk categories (risk-category-based testing suite), and release checkpoints that block updates unless they meet defined safety and performance standards (compatibility gates). Through exploratory validation across multiple LLM versions, we provide evidence that targeted testing in specific risk areas can uncover performance regressions that overall metrics miss. We also identify several open research challenges, including how to systematically build effective test suites, how to set reliable performance thresholds in non-deterministic systems, and how to detect and explain model drift when providers offer limited transparency. Overall, we frame LLM update management as a software supply chain governance problem and outline a research agenda for putting deployer-side compatibility controls into practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。