arXiv:2607.17568cs.LGcs.AI2026-07被引 1

不训练不微调,联合剪枝注意力与前馈层单元。

CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning

论文配图:CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning
图 1 · 摘自论文原文
  • 基于二阶泰勒展开计算单元间耦合损伤,识别协同重要性。
  • 仅需一次前向传播,用矩阵乘积还原全部单元交互关系。
  • 适合资源受限场景的高效结构化模型压缩,无需标签或恢复。

结构化剪枝通过移除整个计算单元(如注意力头和前馈通道组)压缩大语言模型。现有无训练方法多独立评分,隐含假设剪枝损失为各单元损失之和,但Transformer子层通过共享残差流耦合,单独弱的单元可能共同不可或缺。本文提出CoCurve(跨模块联合剪枝曲率),一种仅需校准、无需微调的方法,联合剪枝注意力与前馈单元。对冻结模型与其掩码副本间的词元级KL散度进行二阶泰勒展开,得到单一费舍尔矩阵:对角线为经典节点显著性,非对角线为联合剪枝曲率边——同时移除两单元的额外损伤。在单次消融加性近似下,该矩阵可由M次前向传播的单单元消融特征的格拉姆积重构,无需成对扫描或梯度计算。剪枝转化为单一预算约束的二次规划问题,在共享注意力-前馈预算下一次性求解,无需标签、微调或恢复。

原文摘要 · Abstract (English)

Structured pruning compresses large language models (LLMs) by removing whole computational units, such as attention heads and feed-forward (FFN) channel groups. Most training-free methods, however, rank these units independently, implicitly treating the loss from pruning a set as the sum of its individual losses. This view fails for Transformers, whose sublayers are coupled through a shared residual stream. Two individually weak units can thus be jointly indispensable, yet independent scoring is blind to such dependence and removes them together. We introduce CoCurve (Cross-Module Co-Pruning Curvature), a calibration-only, fine-tuning-free method that prunes attention and FFN units jointly. A second-order Taylor expansion of the token-level KL between the frozen model and its masked copy yields a single Fisher matrix whose diagonal is classical node saliency and whose off-diagonal entries are co-pruning curvature edges: the extra damage of removing two units together. Under a single-ablation additivity approximation this matrix reduces to a Gram product of single-unit ablation features, so the full M x M interaction is recovered from M forward passes, with no pairwise sweeps or gradients. Pruning then reduces to one budgeted quadratic program, solved in a single shot under a shared attention--FFN budget, with no labels, fine-tuning, or recovery.

结构化剪枝零训练联合剪枝大模型压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。