用单个词的对数概率监测大模型接口，低成本发现微小更新
Log Probability Tracking of LLM APIs
- 通过单个输出词的平均对数概率做统计检验，实现高效监控
- 能检测到仅一次微调带来的变化，灵敏度高于现有方法
- 适合关注模型稳定性的研究者和生产系统运维人员
使用大语言模型API时，用户期望服务模型在时间上保持一致，这对下游应用的可靠性与研究可复现性至关重要。现有审计方法成本过高，难以定期应用于众多可用的LLM API，导致模型更新在实践中基本处于无监控状态。本文表明，尽管LLM的对数概率通常具有非确定性，但仍可用于低成本的持续监控。我们基于每个词的对数概率平均值设计了一种简单统计检验，仅需请求一个输出词即可完成检测，灵敏度可达识别一次微调级别的变化，且成本仅为现有方法的1/1000。为此，我们引入TinyChange基准，用于衡量审计方法在真实小规模模型变更下的敏感性。
原文摘要 · Abstract (English)
When using an LLM through an API provider, users expect the served model to remain consistent over time, a property crucial for the reliability of downstream applications and the reproducibility of research. Existing audit methods are too costly to apply at regular time intervals to the wide range of available LLM APIs. This means that model updates are left largely unmonitored in practice. In this work, we show that while LLM log probabilities (logprobs) are usually non-deterministic, they can still be used as the basis for cost-effective continuous monitoring of LLM APIs. We apply a simple statistical test based on the average value of each token logprob, requesting only a single token of output. This is enough to detect changes as small as one step of fine-tuning, making this approach more sensitive than existing methods while being 1,000x cheaper. We introduce the TinyChange benchmark as a way to measure the sensitivity of audit methods in the context of small, realistic model changes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。