通过智能合并层提升大模型压缩效率,质量损失更小且稳定。
XMerge: Cross-Axis Selection and Reconstructive Layer Merging for LLM Depth Compression
- 按隐藏状态变化选层,再重建相邻层输出,减少信息丢失。
- 在7个模型上,4层压缩时6个表现最佳,困惑度增长远低于对手。
- 无需微调或额外参数,适合部署时快速优化模型深度。
移除完整Transformer层可保持标准推理架构,但现有深度压缩方法常导致显著质量下降,且损失不可预测。本文提出XMerge,一种无需任务标签或端到端微调的后训练方法,包含两个组件:跨轴选择识别相对幅度和隐状态角度变化均较低的块,局部边界重建则重新拟合相邻存活块以匹配原两块输出。该方法不引入架构变更或额外推理参数。在七个Llama与Qwen骨干模型(0.5B-8B)、五个公开基线及三种层删减水平下,其优势在最激进压缩时最大:当k=4时,在七模型中六款于CORE(22任务综合)排名第一,另六款于MMLU领先(五款两者并列第一),且避免了多个竞争方法的大幅困惑度上升。任务级自助法显示,三个最大CORE差距的95%置信区间均不包含零;其余差距与平局一致。在14个(模型,策略)组合中,它是唯一未崩溃的评估算子,零样本与上下文学习场景均位列前二;在单骨干校准测试中为最优校准方法。消融实验表明,局部重建贡献主要增益,跨轴融合在两选择轴冲突时有效。额外构建成本在数万次请求后即被每令牌解码节省所回收。
原文摘要 · Abstract (English)
Removing complete transformer layers preserves a standard serving architecture, but existing depth-compression methods can lose substantial quality, and the loss varies unpredictably across models. We introduce XMerge, a post-training method with two components. Cross-axis selection identifies a block with low relative-magnitude and angular hidden-state change, and local boundary reconstruction re-fits the adjacent surviving block to match the original two-block output. XMerge uses no task labels or end-to-end fine-tuning, and it introduces neither architectural changes nor additional inference-time parameters. Across seven Llama and Qwen backbones (0.5B-8B), five published baselines, and three layer-reduction levels, its advantage over baselines is largest at the most aggressive removal: at k=4 it ranks first on six of seven backbones on CORE (a 22-task aggregate) and, separately, on six of seven on MMLU (five of seven on both at once), while avoiding the large perplexity increases of several competing operators. In a task-level bootstrap, the 95% confidence intervals for the three largest CORE margins exclude zero; the remaining margins are consistent with ties. Across the 14 (model, regime) cells it is also the only evaluated operator that never collapses, ranking top-2 in both zero-shot and in-context regimes; on a first calibration probe (one backbone) it is the best-calibrated operator. Ablations show that local reconstruction provides most of the gain, while cross-axis fusion helps when the two selection axes disagree. The additional construction cost is recovered through per-token decode savings after roughly tens of thousands of requests.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。