发现语言模型对齐主要集中在中间层，而非全模型参数调整。
Alignment is Localized: A Causal Probe into Preference Layers
- 通过因果修补法对比基线与调优模型的各层激活
- 仅中间层激活变化显著影响奖励一致性行为
- 适合关注对齐机制原理的研究者阅读
强化学习框架，尤其是基于人类标注的偏好微调方法，已成为对齐语言模型行为的重要手段。其中，基于人类反馈的强化学习（RLHF）尤为流行，用于使模型符合安全性和人类意图。然而，这种对齐的具体内部机制仍不清晰。本文通过在基础模型与调优模型之间对人类偏好样本进行逐层因果修补，系统分析了语言模型的偏好优化过程。实验以 extit{Llama-3.2-1B} 为对象，发现对齐效果在空间上高度集中：仅有中层激活形成一个可因果决定奖励一致行为的独立子空间，而早期和晚期层基本不受影响。利用 LASSO 回归进一步验证，仅有少数几层的激活距离与奖励增益存在非零关联。结果表明，至少在某些语言模型中，基于人类偏好的对齐是一种方向性、低秩的过程，而非广泛分布的参数化调整。
原文摘要 · Abstract (English)
Reinforcement Learning frameworks, particularly those utilizing human annotations, have become an increasingly popular method for preference fine-tuning, where the outputs of a language model are tuned to match a certain set of behavioral policies or guidelines. Reinforcement Learning through Human Feedback (RLHF) is perhaps the most popular implementation of such a framework, particularly for aligning LMs toward safety and human intent. However, the internal workings of how such alignment is achieved remain largely opaque. In this work, we systematically analyze preference optimization for language model alignment by applying layer-wide causal patching between a base model and its tuned counterpart across human preference pairs. We implement our methodology on \textit{Llama-3.2-1B}, and find that alignment is spatially localized: mid-layer activations encode a distinct subspace that causally determines reward-consistent behavior, while early and late layers remain largely unaffected. Utilizing LASSO regression, we also find that only a small number of layers possess non-zero coefficients linking activation distances to reward gains. Overall, we show that, at least for some language models, alignment from human-based, preferential tuning is a directional, low rank process, rather than diffuse and parameteric.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。