arXiv:2609.06974cs.CLcs.AI2026-09

通过训练时过参数化,实现大模型结构化剪枝后的高性能恢复。

Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning

论文配图:Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning
图 1 · 摘自论文原文
  • 训练时临时扩大恢复模块容量,吸收原模型复杂知识
  • 在25%和50%剪枝下,推理性能比现有方法高5.5和8.4分
  • 适合需要高效部署大模型的场景,如边缘设备

大语言模型在多种任务中表现优异,但部署成本高,受内存、延迟和能耗限制。结构化剪枝通过移除模型组件降低成本,但其恢复阶段常受限于恢复模块表征能力与被移除知识复杂度之间的不匹配,称为容量-知识不对称。为此,本文提出OverRep——一种过参数化重参数框架,遵循“训练时过参数化,部署时紧凑”的原则。训练期间,恢复模块被临时过参数化以吸收来自原始模型的复杂知识;恢复后,过参数化部分通过代数合并转化为数学等价的紧凑模块,保持剪枝后模型的推理架构和计算开销。OverRep还引入退火激活机制,在训练中实现非线性动态,最终收敛至线性状态以实现精确代数合并。在三种骨干模型族上,相较于强基线,OverRep在25%和50%剪枝率下分别提升推理性能5.5和8.4分,同时内存占用和TFLOPs与现有方法相当。代码已开源。

原文摘要 · Abstract (English)

Large language models achieve strong performance across diverse tasks, but deployment remains costly because of memory, latency, and energy demands. Structured pruning reduces these costs by removing architectural components, yet its recovery stage is often limited by a mismatch between the recovery module's representational capacity and the complexity of the removed knowledge. We call this bottleneck the capacity-knowledge asymmetry and propose OverRep, an Overcomplete Reparameterization framework for structured LLM pruning. Following the principle of "train overcomplete, deploy compact", OverRep temporarily overparameterizes the recovery module during training to absorb complex knowledge distilled from the original model. After recovery, the overcomplete re-parameterization is algebraically merged into a mathematically equivalent compact module, preserving the pruned model's inference-time architecture and computational cost. OverRep further introduces an annealed activation that enables nonlinear training dynamics while converging to a linear regime for exact algebraic merging. Across three backbone families, OverRep improves retained reasoning performance over strong recovery baselines by up to 5.5 and 8.4 points at 25% and 50% pruning, respectively, while keeping memory usage and TFLOPs comparable to existing recovery methods. Our code is available at https://github.com/mmai-laboratory/OverRep.

大模型剪枝过参数化高效推理模型压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。