通过统计检验检测API模型是否被修改，10个样本即可发现77%的模型变化
Model Equality Testing: Which Model Is This API Serving?
- 用最大均值差异法对比API输出与原始模型分布
- 仅需平均10个样本/提示，对多种修改的检出率达77.4%
- 实测发现超三分之一商用API提供不同模型分布
用户常通过黑盒推理API使用大语言模型，包括闭源和开源模型（如Llama系列可通过Amazon Bedrock和Azure AI Studio访问）。为降低成本或增加功能，API服务商可能对底层模型进行量化、水印或微调，改变输出分布，且未必告知用户。本文将检测此类变化形式化为模型等价性检验，即两样本检验问题：用户从API获取样本，并与参考分布进行统计检验以判断两者是否相同。研究发现，基于最大均值差异（Maximum Mean Discrepancy）的测试在此任务中表现优异；采用简单字符串核的测试在面对多种模型修改时，平均仅需每提示10个样本，即可达到77.4%的检出率（幂）。随后，我们将该测试应用于2024年夏季的四个Llama模型的商业推理API，结果发现31个接口中有11个提供的分布与Meta发布的参考权重不同。
原文摘要 · Abstract (English)
Users often interact with large language models through black-box inference APIs, both for closed- and open-weight models (e.g., Llama models are popularly accessed via Amazon Bedrock and Azure AI Studio). In order to cut costs or add functionality, API providers may quantize, watermark, or finetune the underlying model, changing the output distribution -- possibly without notifying users. We formalize detecting such distortions as Model Equality Testing, a two-sample testing problem, where the user collects samples from the API and a reference distribution and conducts a statistical test to see if the two distributions are the same. We find that tests based on the Maximum Mean Discrepancy between distributions are powerful for this task: a test built on a simple string kernel achieves a median of 77.4% power against a range of distortions, using an average of just 10 samples per prompt. We then apply this test to commercial inference APIs from Summer 2024 for four Llama models, finding that 11 out of 31 endpoints serve different distributions than reference weights released by Meta.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。