将密集模型转为高效设备端MoE模型，兼顾速度与精度
Dense2MoE: Pushing the Pareto Frontier of On-Device LLMs via Unified Pruning and Upcycling
- 通过层融合技术统一剪枝与升维，优化模型结构
- 在保持精度前提下，显著降低推理延迟和参数量
- 适合资源受限设备部署，尤其适合轻量化大模型
混合专家（MoE）架构在资源受限的设备端部署中极具潜力，但从头训练成本过高。现有方法尝试将密集模型升维为MoE，却常引入冗余参数，降低推理效率；而标准层剪枝虽减少冗余，却不可避免损害模型准确率。为此，我们提出Dense2MoE框架，通过层融合升维（LF UC）机制，结合硬件屋顶线理论，系统性地突破推理内存瓶颈：从冗余层中剪除高带宽消耗的注意力模块，同时将其多层感知机（MLP）重用于构建MoE专家。该结构创新在保留模型核心能力的同时，通过选择性令牌路由严格控制活跃参数。仅需少量持续预训练预算，Dense2MoE即可高效将公开的密集大模型转化为设备就绪的MoE模型。大量实验表明，Dense2MoE显著推进了设备端推理延迟与模型精度之间的帕累托前沿，优于密集基线、先进压缩方法及标准升维方案。
原文摘要 · Abstract (English)
The Mixture of Experts MoE architecture is highly promising for resource constrained on device deployments yet training these models from scratch incurs prohibitive costs Current methods attempt to alleviate this by upcycling dense models into MoEs however they often introduce parameter redundancy that degrades inference efficiency Alternatively standard layer pruning mitigates redundancy but inevitably compromises model accuracy To resolve this dilemma we propose Dense2MoE a novel framework that unifies pruning and upcycling through Layer Fusion UpCycling LF UC Guided by hardware Roofline theory Dense2MoE systematically overcomes the inference memory wall by pruning bandwidth heavy attention modules from redundant layers while repurposing their Multi Layer Perceptrons MLPs into MoE experts This structural innovation preserves the models core capabilities and strictly limits active parameters via selective token routing With a modest continual pre training budget Dense2MoE efficiently converts publicly available dense LLMs into on device ready MoE models Extensive experiments demonstrate that Dense2MoE significantly advances the Pareto frontier for on device inference latency versus model accuracy outperforming dense baselines state of the art compression and standard upcycling methods
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。