PrunePath让语言模型的前馈层更高效，稀疏化后仍能快速推理。
PrunePath: Towards Highly Structured Sparse Language Models
- 用概率预算动态激活重要专家，实现灵活稀疏
- 在多种任务上比现有方法更省计算且保持性能
- 支持实际部署，加速解码并节省内存
前馈网络（FFN）占现代语言模型参数和计算量的主导地位，但现有剪枝方法难以将稀疏性转化为硬件友好的推理效率提升。我们提出 extbf{PrunePath}，一种针对FFN层的预算自适应结构化稀疏化框架。基于MoEfication，PrunePath将独立专家阈值替换为软最大化路由分布，并在累积质量阈值下激活重要专家。该设计引入了逐令牌概率预算，实现自适应专家数量，并通过单一检查点提供直接的推理时稀疏控制。在自然语言理解、自然语言生成及指令微调任务中，PrunePath在稀疏性与性能之间取得了优于现有静态剪枝和基于MoEfication方法的权衡。我们进一步实现了Triton内核用于键值缓存解码，将所得结构稀疏性转化为实际内存节省和可测量的解码速度提升。结果表明，PrunePath在构建高度稀疏、可部署的大语言模型方面具有显著优势。
原文摘要 · Abstract (English)
Feed-forward networks (FFNs) dominate the parameter count and computation of modern language models, yet existing pruning methods often struggle to convert sparsity into hardware-friendly inference efficiency gains. We introduce \textbf{PrunePath}, a budget-adaptive structured sparsification framework for FFN layers. Built on MoEfication, PrunePath replaces independent expert-wise thresholding with a softmax-normalized routing distribution and activates important experts under a cumulative-mass threshold. This formulation imposes a token-level probability budget, enabling adaptive expert counts and a direct inference-time sparsity knob from a single checkpoint. Across NLU, NLG, and instruction-tuning evaluations, PrunePath achieves a favorable sparsity--performance trade-off compared with existing static pruning and MoEfication-based methods. We further implement Triton kernels for KV-cache decoding to translate the resulting structured sparsity into practical memory savings and measurable decoding-speed improvements. These results demonstrate the superior performance of PrunePath for building highly sparse, deployment-friendly large language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。