通过对数似然差设计数据域混合，让语言模型更贴近目标模型。
Domain Mixture Design via Log-Likelihood Differences for Aligning Language Models with a Target Model
- 将模型视为对数似然空间中的点，按方向调整训练数据域权重。
- 在NanoGPT上使基模型与目标模型的KL散度降低，优于原始Pile权重。
- 无需知识蒸馏即可实现有效对齐，适合资源有限的场景。
不直接进行知识蒸馏，本文提出一种通过设计预训练或持续预训练的数据域混合来对齐基础模型与目标模型分布的方法。该方法将模型视作对数似然空间中的点，通过使训练更新方向指向目标模型，确定各数据域的权重。在NanoGPT上的实验表明，相比使用Pile原始权重训练，该方法能持续降低训练后基模型与目标模型之间的KL散度。尽管知识蒸馏在可用时仍更有效，但本方法实现了有意义的对齐，下游任务性能也趋于接近目标模型。
原文摘要 · Abstract (English)
Instead of directly distilling a language model, this study addresses the problem of aligning a base model with a target model in distribution by designing the domain mixture of training data for pretraining or continued pretraining as a fixed training recipe. We propose a method for determining domain weights by viewing models as points in log-likelihood space and aligning the training update direction with the direction toward the target model. Experiments with NanoGPT show that the proposed method consistently reduces the KL divergence between the trained base model and the target model relative to training with Pile-original weighting. Although knowledge distillation remains more effective when available, the proposed method achieves meaningful alignment, and downstream task performance also tends to become closer to that of the target model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。