解决大模型小模型词汇不匹配问题，提升小模型性能。
Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling
- 通过词粒度对齐和教师损失引导，跨词汇训练小模型。
- 在1B小模型上，性能比直接微调提升46%。
- 适用于不同词汇的师生模型，尤其适合资源受限场景。
利用大模型指导小模型训练已成为高效学习的主流范式。然而，师生模型间词汇不匹配导致词元序列与输出分布不一致，带来显著挑战。为此，我们提出词汇无关的教师引导语言建模（VocAgnoLM），通过两项关键方法克服此问题：(1) 词粒度词汇对齐，实现跨不匹配词汇的词元序列对齐；(2) 教师引导损失，利用教师模型的损失信号指导学生模型有效训练。我们在使用多个7B教师模型（含不同词汇）时，验证了1B学生模型的语言建模效果。值得注意的是，当教师模型Qwen2.5-Math-Instruct与TinyLlama仅共享约6%词汇时，VocAgnoLM相比朴素持续预训练实现46%的性能提升。此外，实验表明VocAgnoLM始终受益于更强的教师模型，为语言建模中的词汇不匹配问题提供稳健解决方案。
原文摘要 · Abstract (English)
Using large teacher models to guide the training of smaller student models has become the prevailing paradigm for efficient and effective learning. However, vocabulary mismatches between teacher and student language models pose significant challenges in language modeling, resulting in divergent token sequences and output distributions. To overcome these limitations, we propose Vocabulary-agnostic Teacher Guided Language Modeling (VocAgnoLM), a novel approach that bridges the gap caused by vocabulary mismatch through two key methods: (1) Token-level Lexical Alignment, which aligns token sequences across mismatched vocabularies, and (2) Teacher Guided Loss, which leverages the loss of teacher model to guide effective student training. We demonstrate its effectiveness in language modeling with 1B student model using various 7B teacher models with different vocabularies. Notably, with Qwen2.5-Math-Instruct, a teacher model sharing only about 6% of its vocabulary with TinyLlama, VocAgnoLM achieves a 46% performance improvement compared to naive continual pretraining. Furthermore, we demonstrate that VocAgnoLM consistently benefits from stronger teacher models, providing a robust solution to vocabulary mismatches in language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。