DPO通过微调激活向量让模型学会表现得更符合偏好，而非改变其内在信念。
D-STEER - Preference Alignment Techniques Learn to Behave, not to Believe -- Beneath the Surface, DPO as Steering Vector Perturbation in Activation Space
- DPO仅对最终隐藏表示进行一阶微调，不重构语义结构。
- 添加提取的控制向量可复现多数对齐行为，移除则几乎恢复原模型。
- 高层特征空间中存在秩一主导与熵坍缩，表明对齐聚焦于少数方向。
直接偏好优化（DPO）已成为对齐大语言模型的标准方法，但其在模型内部引发的具体变化仍不明确。本文认为，DPO并未改写模型的内在信念，而是作为一种低秩控制机制，沿着少数偏好方向微调激活。通过简单推导，我们发现DPO梯度仅依赖于偏好与非偏好生成结果的逻辑嵌入差异，暗示其仅引起最终隐藏表示的一阶变化，而非深层语义重构。随后，我们从经DPO训练的模型中提取出经验性控制向量，并证明将该向量加至基础激活上可重现大部分对齐行为，而减去它则近乎恢复原始模型。最后，谱分析显示高层特征空间呈现秩一主导和熵坍缩，表明对齐被限制在狭窄子空间中。综合来看，这些结果支持一种行为幻觉观点：DPO教会模型如何表现得对齐，而非相信什么。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) has become a standard recipe for aligning large language models, yet it is still unclear what kind of change it actually induces inside the network. This paper argues that DPO does not rewrite a models internal beliefs; instead, it acts as a low rank steering mechanism that nudges activations along a small number of preference directions. Using a simple derivation, we show that the DPO gradient depends only on the difference between the logit embeddings of preferred and dispreferred completions, implying a first order shift in the final hidden representation rather than a deep restructuring of semantics. We then extract an empirical steering vector from a DPO tuned model and demonstrate that adding this vector to base activations reproduces most of the aligned behavior, while subtracting it nearly restores the original model. Finally, spectral analyses reveal rank-one dominance and entropy collapse in upper layers, indicating that alignment is funneled through a narrow subspace. Taken together, these results support a behavioral illusion view of DPO: it teaches models how to act aligned, not what to believe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。