通过行为指纹检测大模型接口的隐性变化，保障AI应用稳定性。
Behavioral Fingerprints for LLM Endpoint Stability and Identity
- 用固定提示词采样输出构建行为指纹，对比分布变化
- 检测到模型版本、量化、推理栈等变更，准确率100%
- 适合关注模型部署稳定性的工程师和运维团队
AI原生应用的稳定性依赖于其背后模型接口的行为一致性。传统可靠性指标如可用性、延迟和吞吐量无法捕捉行为变化，一个接口可能保持“健康”状态，但因权重更新、分词器调整、量化、推理引擎、内核、缓存、路由或硬件变更而导致实际模型身份改变。我们提出Stability Monitor，一种黑盒稳定性监控系统，通过定期对固定提示集采样输出并比较其输出分布随时间的变化来生成行为指纹。使用总能量距离统计量跨提示词比较指纹，结合置换检验的p值作为分布偏移证据，并逐次聚合以检测变化事件并定义稳定期。在受控验证中，该系统成功检测到模型家族、版本、推理栈、量化方式及行为参数的更改。在多个提供商托管同一模型的真实场景中，观察到显著的提供商间及同一提供商内部的稳定性差异。
原文摘要 · Abstract (English)
The consistency of AI-native applications depends on the behavioral consistency of the model endpoints that power them. Traditional reliability metrics such as uptime, latency and throughput do not capture behavioral change, and an endpoint can remain "healthy" while its effective model identity changes due to updates to weights, tokenizers, quantization, inference engines, kernels, caching, routing, or hardware. We introduce Stability Monitor, a black-box stability monitoring system that periodically fingerprints an endpoint by sampling outputs from a fixed prompt set and comparing the resulting output distributions over time. Fingerprints are compared using a summed energy distance statistic across prompts, with permutation-test p-values as evidence of distribution shift aggregated sequentially to detect change events and define stability periods. In controlled validation, Stability Monitor detects changes to model family, version, inference stack, quantization, and behavioral parameters. In real-world monitoring of the same model hosted by multiple providers, we observe substantial provider-to-provider and within-provider stability differences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。