用低秩投影让小模型高效复现大模型能力，训练效率提升千倍以上。
A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone
- 通过低秩投影实现软剪枝与激活对齐一体化。
- 仅用200亿token训练即超越万亿级数据训练的SOTA模型。
- 适合追求高性价比小模型训练的研究者与开发者。
训练高性能小语言模型（SLMs）仍成本高昂，即使经过大模型教师的蒸馏与剪枝。现有方法常面临三大挑战：(1) 硬剪枝导致的信息丢失，(2) 表示对齐效率低下，(3) 尤其是前馈网络（FFN）中信息激活的利用率不足。为此，我们提出低秩克隆（LRC），一种高效的预训练方法，旨在使小模型在行为上等同于强教师模型。LRC训练一组低秩投影矩阵，联合实现教师权重压缩（软剪枝）和学生激活对齐（包括FFN信号），统一设计最大化知识迁移，无需额外对齐模块。大量实验表明，在开源教师模型（如 Llama-3.2-3B-Instruct、Qwen2.5-3B/7B-Instruct）上，LRC仅使用200亿训练样本，即可达到或超越使用万亿级样本训练的先进模型性能，实现超过1,000倍的训练效率提升。代码与模型检查点见 https://github.com/CURRENTF/LowRankClone 及 https://huggingface.co/collections/JitaiHao/low-rank-clone-lrc-6828389e96a93f1d4219dfaf。
原文摘要 · Abstract (English)
Training high-performing Small Language Models (SLMs) remains costly, even with knowledge distillation and pruning from larger teacher models. Existing work often faces three key challenges: (1) information loss from hard pruning, (2) inefficient alignment of representations, and (3) underutilization of informative activations, particularly from Feed-Forward Networks (FFNs). To address these challenges, we introduce Low-Rank Clone (LRC), an efficient pre-training method that constructs SLMs aspiring to behavioral equivalence with strong teacher models. LRC trains a set of low-rank projection matrices that jointly enable soft pruning by compressing teacher weights, and activation clone by aligning student activations, including FFN signals, with those of the teacher. This unified design maximizes knowledge transfer while removing the need for explicit alignment modules. Extensive experiments with open-source teachers (e.g., Llama-3.2-3B-Instruct, Qwen2.5-3B/7B-Instruct) show that LRC matches or surpasses state-of-the-art models trained on trillions of tokens--while using only 20B tokens, achieving over 1,000x training efficiency. Our codes and model checkpoints are available at https://github.com/CURRENTF/LowRankClone and https://huggingface.co/collections/JitaiHao/low-rank-clone-lrc-6828389e96a93f1d4219dfaf.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。