通过权重空间分析，逆向推断大模型训练时的数据混合比例。
WARP: Weight-Space Analysis for Recovering Training Data Portfolios
- 利用模型合并生成伪检查点，还原训练轨迹的几何特征。
- 在BERT和GPT-2上平均绝对误差低至0.046和0.104。
- 适合研究模型训练数据构成的学者，可破解数据配方黑箱。
基础模型常被公开发布，但其训练所用的数据配方——如不同数据源的采样权重——极少披露。这导致信息不对称：研究者只能分析模型性能，却无法了解训练分布。现有方法如成员推理仅能识别单个样本，无法刻画训练语料的全局组成。本文提出WARP框架，直接从发布的模型权重中恢复微调模型的训练混合比例。该方法通过模型合并在基线与微调模型间插值，生成近似训练轨迹的伪检查点，揭示权重空间中的训练数据几何足迹。从中提取几何特征，并使用无参softmax或基于合成混合数据训练的MLP投影器映射为领域比例。在BERT和GPT-2的受控实验中，平均绝对误差分别低至0.046和0.104，优于成员推理及具有真实训练轨迹访问权限的变体。
原文摘要 · Abstract (English)
Foundation models are routinely released to the public, yet the data recipes used to train them -- such as domain mixture weights that determine how different sources are sampled -- are rarely disclosed. This creates an access asymmetry: researchers study the resulting models but lack visibility into the training distribution that produces them. Prior works for inferring training data, such as membership inference, detect at the level of individual samples and thus cannot characterize the global composition of the training corpus. We introduce WARP, a framework that recovers a fine-tuned model's training mixtures directly from its released weights. WARP interpolates between the base and fine-tuned models using model merging, generating pseudo-checkpoints that approximate the missing training trajectory and expose a geometric footprint of the training data in the weight space. From these simulated footprints, WARP extracts geometric features and maps them to domain proportions using either a parameter-free softmax readout or an MLP projector trained on synthetic mixtures. In controlled experiments with BERT and GPT-2, WARP recovers domain mixtures with an average MAE as low as 0.046 and 0.104 respectively, outperforming membership inference and a variant with access to the true training trajectory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。