提出新方法提升压缩模型的推理能力，避免因参数剪枝导致的推理失效。
Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization
- 通过激活感知初始化，将投影矩阵设计为通道选择结构
- 实验显示在Llama和Qwen上恢复了严重下降的多步推理能力
- 保持高效训练的同时，兼顾通用能力和推理性能
高效蒸馏（EDistill）通过结构化剪枝参数并微调轻量模块来压缩大语言模型（LLMs），虽在通用能力基准上达到同类模型最优表现，但其多步推理能力显著下降，称为推理崩溃。我们系统分析发现，基于宽度缩减投影矩阵的SOTA方法存在有效秩（eRank）崩溃问题，即隐藏表示的有效秩降低。理论分析表明，随机初始化的投影矩阵奇异值分布不均，导致eRank崩溃和标记不可区分。为此，我们提出RED（Reasoning-preserved Efficient Distillation）方法，引入激活感知初始化，使投影矩阵成为通道选择矩阵，理论上缓解eRank崩溃。在Llama和Qwen系列模型上的实验表明，RED显著恢复推理能力，同时保持高训练效率与最优通用性能。
原文摘要 · Abstract (English)
Efficient Distillation (EDistill) compresses large language models (LLMs) by structured pruning parameters and tuning lightweight modules with high training efficiency. Although these EDistilled LLMs achieve state-of-the-art (SOTA) performance on general ability benchmarks relative to similarly sized LLMs, we identify a severe degradation in their multi-step reasoning ability, which we term reasoning collapse. We systematically analyze the geometric origins of reasoning collapse and show that the SOTA EDistill method based on width-reducing projection matrices suffers from eRank collapse, in which the effective rank (eRank) of hidden representations drops. We theoretically explain how singular values of randomly initialized projection matrices become unevenly distributed, leading to eRank collapse and thus token indistinguishability. To address this issue, we propose RED (Reasoning-preserved Efficient Distillation) for LLMs, which introduces activation-aware initialization to initialize projection matrices as channel-selection matrices, thus theoretically mitigating eRank collapse. Experiments on Llama and Qwen series demonstrate that RED substantially recovers reasoning while maintaining high training efficiency and SOTA general ability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。