语言模型内部会追踪自身策略是否正确，影响其自信与纠错行为。
The Value Axis: Language Models Encode Whether They're on the Right Track
- 构建价值轴衡量模型对目标达成的预期信心
- 高价值时减少纠错、降低冗余解释，低价值时触发回溯探索
- 适合研究模型决策机制与安全对齐的读者
我们探究语言模型是否内部分析当前策略达成目标的可能性。基于合成的上下文强化学习数据，为 Qwen3-8B 构建了「价值」轴。结果发现，该轴上的激活值能区分高/低口头自信、有无回溯的推演过程，以及代码正确与否。向高价值方向引导会因果性地抑制自我修正并减少解释冗余；向低价值方向则引发回溯与探索。我们证明直接偏好优化（DPO）可提升被奖励行为的内部价值，使模型在表现出该行为后更自信。进一步应用于真实场景：发现后训练阶段，模型对政治敏感问题赋予低价值；而监督微调增强了训练域内的内部自信。结果表明，语言模型线性编码了对目标成功概率的估计，并据此调节其推进方向的信心程度。
原文摘要 · Abstract (English)
We investigate whether language models internally track the value of their current trajectory, defined as the likelihood that their ongoing strategy will achieve their goals. Using synthetic, in-context reinforcement learning data, we construct a "value" axis for Qwen3-8B. We find that activations along this axis distinguish between high vs. low verbalized confidence, rollouts without and with backtracking, and correct vs. corrupted code. Steering towards high value causally suppresses self-correction and reduces explanatory verbosity, while steering towards low value induces backtracking and exploration. We demonstrate that direct preference optimization (DPO) can increase the internal value of rewarded behaviors (e.g. use a certain word), causing the model to act more confidently after exhibiting them. Finally, we apply the value axis to study in-the-wild settings. For example, we find that Qwen assigns low value to politically sensitive chat queries after post-training and that supervised fine-tuning increases internal confidence within the training domain. Our results suggest that language models linearly encode an estimate of expected goal success that modulates their confidence in pursuing a direction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。