不同测试协议下，变压器层是否可删减结果差异显著，需同时评估替换与互换性。
No Free Swap: Protocol-Dependent Layer Redundancy in Transformers
- 区分替换与互换两种评估协议，揭示层等价性的测量差异
- 在Pythia训练中，协议差距随训练进展扩大，影响剪枝安全度数倍变化
- 建议剪枝前同时测试两种交换KL，无需标签数据
研究者在评估Transformer层是否‘等价’以用于压缩时，常混淆两种不同测试：替换测试关注某一层能否直接替代另一层；互换测试则考察两层位置互换后输出是否近似不变。两者均为基于输出的交换KL探测，但结果未必一致。在预训练模型中，协议差距可能导致同一评估器下可安全剪枝的层数相差数倍，尤其当替换距离较大时。我们在多个检查点和架构上测量了这两种协议。在Pythia训练轨迹（410M和1.4B）中，替换-互换差距从初始化到收敛持续增长。在8B规模的WikiText-2任务下，Qwen3-8B进入发散状态：基于互换引导的剪枝比替换引导的安全性高数倍，而尽管Llama-3.1-8B的互换KL更低，二者剪枝代价仍相当，表明指标差距未必对应实际移除效果。层剪枝或合并前，应在目标检查点上同时计算两种交换KL，该诊断仅需无标签前向传播。
原文摘要 · Abstract (English)
When researchers ask whether two transformer layers are "equivalent" for compression, they often conflate distinct tests. Replacement asks whether one layer's map can substitute for another's in place; interchange asks whether two layers approximately commute when their positions are swapped. Both are output-grounded swap-KL probes, but they need not agree: on pretrained transformers the protocol gap can change which layers look safe to prune by several-fold under the same evaluator, especially when replacement distances are high. We measure both protocols across checkpoints and architectures. On a Pythia training trajectory (410M and 1.4B), the replacement-interchange gap grows from initialization to convergence. Under one matched WikiText-2 contract at 8B scale, Qwen3-8B enters a divergent regime: interchange-guided removal is several-fold safer than replacement-guided at the same layer budgets, while Llama-3.1-8B ties the two protocols for pruning cost even though interchange KL is lower, showing metric gaps need not map one-to-one to removal. Before layer removal or merging, score both swap-KLs on the target checkpoint; the diagnostic requires only unlabeled forward passes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。