用随机初始化差异追踪大模型训练源头,从零开始就能识别模型出身
SeedPrints: Fingerprints Can Even Tell Which Seed Your Large Language Model Was Trained From
- 利用模型初始化种子带来的固有偏差作为唯一指纹
- 即使未训练的模型也能通过预测偏见区分不同训练种子
- 适用于预训练全周期,对数据分布变化和参数修改都鲁棒
大型语言模型(LLM)的指纹技术对模型溯源与归属验证至关重要。现有方法主要在微调后评估,依赖训练过程中形成的稳定签名,但模型大部分知识来自预训练阶段,使预训练成为更根本的溯源场景。我们发现,现有方法在此阶段失效,因其依赖训练后才出现的后验特征。这违背了指纹作为内在持久身份的经典定义。为此,我们提出更本质的指纹概念——SeedPrints,利用随机初始化偏差作为训练前就存在的、种子相关的标识符。我们证明未训练模型即存在由初始化种子引发的可复现预测偏差,且该弱信号在整个训练过程保持统计可检测性,实现高置信度的模型谱系追溯。相比以往方法在早期预训练中失效或在分布漂移下退化,SeedPrints在从初始化到大规模预训练及下游适配的全阶段均有效。在基于LLaMA和Qwen风格模型的实验中,实现了种子级可区分性,支持从出生到生命周期的身份验证。大规模预训练轨迹和真实指纹基准测试进一步验证其在长期训练、领域迁移和参数修改下的鲁棒性。
原文摘要 · Abstract (English)
Fingerprinting Large Language Models (LLMs)is essential for provenance verification and model attribution. Existing fingerprinting methods are primarily evaluated after fine-tuning, where models have already acquired stable signatures from training data, optimization dynamics, or hyperparameters. However, most of a model's capacity and knowledge are acquired during pretraining rather than downstream fine-tuning, making large-scale pretraining a more fundamental regime for lineage verification. We show that existing fingerprinting methods become unreliable in this regime, as they rely on post-hoc signatures that only emerge after substantial training. This limitation contradicts the classical Galton notion of a fingerprint as an intrinsic and persistent identity. In contrast, we propose a stronger and more intrinsic notion of LLM fingerprinting: SeedPrints, a method that leverages random initialization biases as persistent, seed-dependent identifiers present even before training begins. We show that untrained models exhibit reproducible prediction biases induced by their initialization seed, and that these weak signals remain statistically detectable throughout training, enabling high-confidence lineage verification. Unlike prior techniques that fail during early pretraining or degrade under distribution shifts, SeedPrints remains effective across all training stages, from initialization to large-scale pretraining and downstream adaptation. Experiments on LLaMA-style and Qwen-style models demonstrate seed-level distinguishability and enable birth-to-lifecycle identity verification. Evaluations on large-scale pretraining trajectories and real-world fingerprinting benchmarks further confirm its robustness under prolonged training, domain shifts, and parameter modifications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。