通过分析模型输出的词元特征，无需训练即可识别大模型来源与演化关系。
TokenPrint: A Calibrated Token-Space Fingerprint for Language-Model Provenance
- 用固定探针激发隐藏状态，提取词元投影做相似性比对。
- 相同数据训练的模型间相似度达0.48，可区分不同训练背景。
- 适用于模型溯源、版权验证，尤其适合研究者与监管方使用。
确定语言模型的来源（包括基础检查点及训练数据重叠）是治理难题，仅靠元数据无法解决。我们提出一种无需训练的指纹方法，基于250个固定知识探针诱发的晚期隐藏状态的前k个词汇投影，通过解码后词元字符串的杰卡德重叠进行比较。在9个系列共32个开源模型（0.6B–32B）上评估，结果表明：(1) 相似性梯度与模型关联性一致——独立训练于相同数据的模型原始得分0.48（词汇修正后0.35），共享基础微调模型为0.39/0.33，同开发者衍生模型0.38/0.28，无文档关联模型0.22/0.17。该信号在三个组织、两种分词器、两类架构中持续存在，且在训练初期1%内即显现，表明共享数据贡献超越能力收敛。(2) 作为最近邻谱系检索方法，该指纹在五次R1蒸馏任务中均将真实基模型排进前两名（平均排名1.8，MRR 0.60），包括无法从粗粒度元数据识别的数学专用基模型。(3) 深度消融显示谱系区分能力随输出分布增强，AUC从四分之一深度的0.72升至输出层0.90；仅用前5个输出词元仍保持AUC 0.87。(4) 指纹在量化下稳定，int8下杰卡德相似度0.92，int4下0.82–0.85，高于校准池中最大跨模型相似度0.81。我们公开探针、代码和指纹数据。
原文摘要 · Abstract (English)
Establishing the provenance of a language model---including its base checkpoint and possible overlap in training distributions---is a governance challenge that metadata alone cannot resolve. We introduce a training-free fingerprint based on the top-$k$ vocabulary projections of late hidden states elicited by 250 fixed knowledge probes, compared using Jaccard overlap over decoded token strings. We evaluate the method on 32 open-weight models from nine families (0.6B--32B) with documented relationships. (1)~A \emph{similarity ladder} broadly follows model relatedness: independently trained models on identical data score 0.48 raw (0.35 vocabulary-corrected), followed by shared-base fine-tunes (0.39/0.33), same-developer relatives (0.38/0.28), and models with no documented relationship (0.22/0.17). This identical-data signal persists across three organizations, two tokenizer families, and two architecture classes, and emerges within the first 1\% of training before measurable task competence, suggesting a contribution from shared training data beyond capability convergence. (2)~As a nearest-neighbor \emph{lineage-retrieval} method, the fingerprint ranks the exact documented base among the top two candidates for all five R1 distillations (mean rank 1.8, MRR 0.60), including a math-specialized base not identifiable from coarse metadata. (3)~A \emph{depth ablation} shows that lineage group discrimination strengthens toward the output distribution, with AUC increasing from 0.72 at quarter depth to 0.90 at the output; using only the top 5 output tokens retains AUC 0.87. (4)~The fingerprint remains stable under quantization, with Jaccard similarity of 0.92 under int8 and 0.82--0.85 under int4, compared with a maximum cross-model similarity of 0.81 in the calibration pool. We release the probes, code, and fingerprints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。