通过文本特征分布对比，快速检测大模型行为变化。
You've Changed: Detecting Modification of Black-Box Large Language Models
- 比较生成文本的语言与心理语言学特征分布差异。
- 仅用简单特征和统计检验即可区分不同模型。
- 适合关注模型稳定性或安全性的开发者使用。
大型语言模型（LLMs）通常通过API提供服务，导致开发者难以察觉其行为变化。本文提出一种监测方法，通过比较生成文本的语言学与心理语言学特征分布来检测模型变更。该方法利用统计检验判断两组文本特征分布是否等价，从而识别模型是否发生变化。我们以五个OpenAI的补全模型和Meta的Llama 3 70B聊天模型为实验对象，验证了该方法的有效性。结果表明，结合简单文本特征与统计检验可有效区分不同语言模型。此外，我们还探索了该方法在检测提示注入攻击中的应用。本工作支持对LLM进行频繁监控，避免了计算成本高昂的基准评估。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are often provided as a service via an API, making it challenging for developers to detect changes in their behavior. We present an approach to monitor LLMs for changes by comparing the distributions of linguistic and psycholinguistic features of generated text. Our method uses a statistical test to determine whether the distributions of features from two samples of text are equivalent, allowing developers to identify when an LLM has changed. We demonstrate the effectiveness of our approach using five OpenAI completion models and Meta's Llama 3 70B chat model. Our results show that simple text features coupled with a statistical test can distinguish between language models. We also explore the use of our approach to detect prompt injection attacks. Our work enables frequent LLM change monitoring and avoids computationally expensive benchmark evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。