检测大模型是否源自相同初始化,可识别模型间抄袭与剪枝关系。
Independence Tests for Language Models
- 基于权重与激活相似性,设计统计检验方法判断模型训练独立性。
- 在210对模型中准确识别出非独立模型对,即使经过微调仍有效。
- 新方法可定位具体共享组件,且对架构变化和对抗攻击具有鲁棒性。
我们研究一个问题:给定两个模型的权重,能否测试它们是否由独立随机初始化训练而来?在受限设置下,假设模型架构与训练过程一致,提出一组统计检验,可精确计算零假设(即独立初始化)下的p值。这些p值不受任一模型训练数据组成影响,通过模拟满足交换性的模型副本,比较原始模型对与副本间的权重与激活相似性得出。我们在21个开源模型(共210对)上进行测试,正确识别所有非独立模型对,即使其中一个模型经大量文本微调也依然有效。在无约束设置下,不假设训练方式、允许改变架构或对抗扰动时,前述方法失效。为此,提出一种匹配隐藏激活的新测试方法,对对抗变换和架构变化保持鲁棒,并支持局部检测——可定位模型中非独立的部分。尽管无法获得精确p值,但实验表明其表现如同真实检验,能可靠识别非独立模型。尤为关键的是,该方法可识别特定部分的来源,例如Llama 3.1-8B如何剪枝生成Llama 3.2-3B,或Mistral-7B与StripedHyena-7B之间的共享层;即使重训某层也依然有效。
原文摘要 · Abstract (English)
We consider the following problem: given the weights of two models, can we test whether they were trained independently -- i.e., from independent random initializations? We consider two settings: constrained and unconstrained. In the constrained setting, we make assumptions about model architecture and training and propose a family of statistical tests that yield exact p-values with respect to the null hypothesis that the models are trained from independent random initializations. These p-values are valid regardless of the composition of either model's training data; we compute them by simulating exchangeable copies of each model under our assumptions and comparing various similarity measures of weights and activations between the original two models versus these copies. We report the p-values from these tests on pairs of 21 open-weight models (210 total pairs) and correctly identify all pairs of non-independent models. Our tests remain effective even if one model was fine-tuned for many tokens. In the unconstrained setting, where we make no assumptions about training procedures, can change model architecture, and allow for adversarial evasion attacks, the previous tests no longer work. Instead, we propose a new test which matches hidden activations between two models, and which is robust to adversarial transformations and to changes in model architecture. The test can also do localized testing: identifying specific non-independent components of models. Though we no longer obtain exact p-values from this, empirically we find it behaves as one and reliably identifies non-independent models. Notably, we can use the test to identify specific parts of one model that are derived from another (e.g., how Llama 3.1-8B was pruned to initialize Llama 3.2-3B, or shared layers between Mistral-7B and StripedHyena-7B), and it is even robust to retraining individual layers of either model from scratch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。