将多个大小不同的语言模型嵌套训练，提升效率并降低计算成本。
Matryoshka Language Model Suites
- 将不同规模模型嵌套在统一架构中端到端训练
- 相比独立训练，减少36%训练算力，推测解码速度提升14%-26%
- 适合需要高效推理的场景，如推测解码
传统语言模型套件需分别训练并独立部署。本文提出将不同规模的子模型以嵌套方式整合进单一端到端训练架构，形成马特罗什卡训练框架。该方法不仅降低整体参数量，支持每步训练中从大模型向所有小模型低成本蒸馏，还天然适用于推测解码——因草稿模型包含于验证器内。我们训练了一个包含5亿、15亿和30亿参数的马特罗什卡套件，其在基准性能、验证与跨域困惑度上与独立训练基线相当，但训练计算量减少36%，推测解码吞吐率提升14%-26%。同时通过消融实验为构建高效马特罗什卡语言模型套件提供指导。
原文摘要 · Abstract (English)
Training a language model suite classically requires training each model separately and serving them independently. We improve both training and inference efficiency by stacking sub-models of increasing size into a single nested architecture trained end-to-end. This Matryoshka training framework reduces the total parameter count of the suite, enables low-cost distillation from the largest to all smaller sub-models at every training step, and is well-suited for speculative decoding as the draft model is contained within the verifier. We validate our approach by training a Matryoshka suite comprising 500M, 1.5B, and 3B sub-models. Our suite is on par with independently trained baselines on benchmark performance and validation and out-of-domain perplexities, while using 36% less training compute and improving the throughput of speculative decoding by 14-26%. We also ablate key architectural choices, offering guidance for building strong Matryoshka LM suites.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。