用稀疏结构改造大模型，让密集模型变高效稀疏模型。
Continual LLM Upcycling: A Predictor-Gated Bank-Wise Sparsity Training Recipe for Dense-to-Sparse LLMs
- 用低秩预测器动态分配通道，实现每64通道保留16个，4倍稀疏
- 在32K上下文持续训练中保持性能，较原模型仅损失0.8%困惑度
- 适合追求推理效率的工程师，尤其关注长文本处理场景
本文研究从密集模型持续训练生成通道稀疏的大语言模型。基于Qwen2.5-8B密集权重，我们在32K上下文长度下继续训练，并引入预测器门控的稀疏SwiGLU前馈网络（FFN）。每个令牌和层使用低秩预测器生成通道路由得分，再按每64通道保留16个的银行级顶k规则，使FFN中间激活实现4倍稀疏。与后处理稀疏推理不同，路由模块嵌入主语言建模路径并参与持续训练，使密集模型可被“升级”为适配硬件的稀疏模型。实验报告了架构、训练方案、基准表现及调参经验。同时发现RULER-CWE数据集上存在层内长上下文失效问题，提出单层修复算法，显著提升长序列表现。
原文摘要 · Abstract (English)
We study dense-to-sparse continual training as a way to construct channel-sparse large language models from dense checkpoints. Starting from a Qwen2.5-8B dense backbone, we continue training at 32K context and introduce a predictor-gated sparse SwiGLU FFN in the 32K stage. For each token and layer, we use a low-rank predictor to produce FFN-channel routing logits. We then apply a bank-wise top-k rule to retain 16 channels in every 64-channel bank, yielding 4x sparsity in the FFN intermediate activation. Unlike post-hoc sparse inference methods, the routing module is placed on the main language modeling path and optimized during continual training, enabling the dense model to be upcycled into a hardware-oriented sparse model. We report the architecture, training recipe, benchmark performance, and training lessons. We also identify a layer-local long-context failure mode on RULER-CWE and propose a single-layer repair algorithm that substantially improves the affected length range.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。