通过跳过最后几层,让大模型推理更准
Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding

- 动态选择最可靠的近末层进行解码,避免最后一层的干扰
- 在多个推理任务上提升表现,延迟增加不足2%
- 无需训练、零内存开销,适合所有对齐过的语言模型
自回归生成通常从深层表示中解码,假设越深的特征越可靠。我们发现:早期层形成粗略猜测,中间层优化语义推理,而最终层常将预测推向通用或对齐偏好词汇。为此提出无训练解码策略「置信解码」,通过熵引导的保守后向搜索,动态选择最可靠的近末层。理论上将层选择建模为最优停止问题,在投影噪声有界且后期对齐扰动占主导的条件下,该策略可过滤扰动并控制与理想层的损失。在密集型与专家混合(MoE)模型上测试,于GPQA-Diamond、Omni-MATH和HLE等复杂推理任务中实现一致提升,零内存开销,延迟增加低于2%。结果表明,规避末层扰动可释放对齐模型更强的推理能力。
原文摘要 · Abstract (English)
Autoregressive generation in large language models (LLMs) conventionally decodes from the final layer, assuming that deeper representations yield more reliable next-token predictions. We revisit this assumption by revealing a recurring Guess-Refine-Perturb dynamic: early layers form coarse guesses, intermediate layers refine reasoning-relevant semantics, and final layers can perturb these refined predictions toward generic or alignment-preferred tokens. We introduce Confident Decoding, a training-free decoding strategy that dynamically selects the most reliable near-final layer through entropy-guided conservative backward search. We further provide a theoretical formulation of layer selection as an optimal stopping problem, showing that under bounded projection noise and dominant late-stage alignment perturbation, our search rule filters perturbation while bounding the loss relative to the oracle refinement layer. Experiments across dense and Mixture-of-Experts LLMs demonstrate consistent gains on challenging reasoning benchmarks, including GPQA-Diamond, Omni-MATH, and HLE, with zero memory overhead and less than 2% latency increase. These results suggest dynamically bypassing final-layer perturbations can unlock stronger reasoning behavior from aligned LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。