提出无随机性的结构化剪枝方法，显著降低大模型推理开销且性能损失小。
Deterministic Differentiable Structured Pruning for Large Language Models
- 直接优化确定性软代理目标，消除传统方法的随机性
- 在20%稀疏度下性能损失仅1%，优于已有方法
- 适用于Qwen等密集模型与MoE架构，部署效率高
结构化剪枝通过移除低重要性的模型组件来降低大语言模型的推理成本，可视为在l0稀疏约束下为每个组件学习一个乘性门。由于l0范数的离散性，先前工作通常采用随机硬-混凝土松弛以实现可微优化；但这种随机性会导致训练-测试不一致，并限制掩码处于有界近二值范围。为此，我们提出确定性可微剪枝（DDP），一种仅优化掩码的方法，通过直接优化离散l0目标的确定性软代理来消除随机性。相比以往方法，DDP具有更强表达力、更小的训练-测试差异和更快收敛速度。我们将该方法应用于多个稠密模型与MoE模型，包括Qwen3-32B和Qwen3-30B-A3B，在下游任务上性能损失最小可达1%，且在20%稀疏度下超越此前方法。进一步在vLLM中验证了端到端推理加速效果。
原文摘要 · Abstract (English)
Structured pruning reduces LLM inference cost by removing low-importance architectural components. This can be viewed as learning a multiplicative gate for each component under an l0 sparsity constraint. Due to the discreteness of the l0 norm, prior work typically adopts stochastic hard-concrete relaxations to enable differentiable optimization; however, this stochasticity can introduce a train--test mismatch when sampled masks are discretized for deployment and restricts masks to a bounded, near-binary range. To address this, we propose Deterministic Differentiable Pruning (DDP), a mask-only optimization method that eliminates stochasticity by directly optimizing a deterministic soft surrogate of the discrete l0 objective. Compared with prior approaches, DDP offers greater expressiveness, reduced train--test mismatch, and faster convergence. We apply our method to several dense and MoE models, including Qwen3-32B and Qwen3-30B-A3B, achieving a performance loss as small as 1% on downstream tasks while outperforming previous methods at 20% sparsity. We further demonstrate end-to-end inference speedups in realistic deployment settings with vLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。