无需训练，通过近似中间通道重要性实现高效稀疏化。
Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs
- 用输入稀疏性和量化代理权重构建共享通道掩码
- 在70%稀疏度下实现1.99倍端到端解码加速
- 兼容量化和稀疏注意力，适合部署优化场景
前馈网络（FFN）在大语言模型推理中主导了内存流量和计算开销，是激活稀疏化的首要目标。现有无训练方法在高稀疏度下因通道选择策略局限导致模型质量显著下降。我们发现SwiGLU的中间状态提供极强的通道选择信号，但其获取需昂贵的密集计算。为此，提出 extit{Prox}，一种两阶段无训练稀疏化框架。核心洞察：稀疏执行仅需由中间状态诱导的通道掩码，可基于其元素大小排序而非精确值构建。第一阶段利用输入稀疏性和量化代理权重构造共享掩码；第二阶段精确计算选中通道，实现所有三投影的稀疏执行。在六个模型家族的十款LLM上，Prox在所有稀疏度下均优于无训练基线，在70% FFN稀疏度下达到最高1.99倍端到端解码加速，并兼容量化与稀疏注意力。
原文摘要 · Abstract (English)
Feed-forward networks (FFNs) dominate memory traffic and computation in large language model (LLM) inference, making them a primary target for activation sparsification. However, existing training-free methods suffer substantial model-quality degradation at high sparsity due to limitations in their channel-selection strategies. We observe that the SwiGLU intermediate state provides a highly effective channel-selection signal, but obtaining it requires costly dense computation. To address this, we present \emph{Prox}, a two-stage training-free framework for sparse SwiGLU FFNs. Prox hinges on the key insight: sparse execution requires only the channel mask induced by the intermediate state, which can be constructed from the magnitude ranking of its entries rather than their exact values. Specifically, Stage 1 uses input sparsity and quantized proxy weights to construct a shared mask; Stage 2 computes the selected channels exactly, enabling sparse execution of all three projections. Across ten LLMs from six model families, Prox outperforms training-free baselines at all sparsity levels, achieves up to a $1.99\times$ end-to-end decoding speedup at 70\% FFN sparsity, and is compatible with quantization and sparse attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。