通过激活值读取模型内部价值优先级,可提前发现价值观冲突。
Constitutional Value Potentials: reading and steering internal priority margins in language models
- 从隐藏状态中学习每个价值的内在线性势能,通过差值构建优先级边际。
- 监控信号在冲突预测上达到0.95的AUROC,且能提前至首个词时识别风险。
- 适用于检测对抗性提示攻击与干预式价值观引导,适合安全对齐研究者。
协议规定语言模型应重视什么价值,但很少有方法判断它是否真正遵循。现有评估依赖输出,而输出在价值冲突时最不可靠——关键不在于模型提到了哪个价值,而在于它愿意牺牲哪个。我们发现,这种权衡可从结构化激活边际中读取。提出宪法价值势能(CVP):为每个价值学习一个由隐藏状态决定的标量势能,代表内在保护该价值的压力;其符号差构成优先级边际。一条宪法条款即要求该边际保持正值,单一监控分数可标记其失效。该监控在冲突违规预测上达到最高0.95的AUROC,优于强基线隐藏状态探测器,并泛化到三个Qwen2.5规模的未见合成冲突。信号在回答开始时即出现,来自提示末尾和首个响应词。提前读取此信号,可判断对抗性提示是否真正导致价值偏离,而非仅表面可疑。相同方向还可用于干预测试:在特定调参下沿价值方向移动,可使被评判的权衡向预期方向变化。结果表明,部分宪法相关优先级可作为激活空间中的边际被访问,而不仅限于输出行为。
原文摘要 · Abstract (English)
A constitution tells a language model what to value, but little tells us whether it does. Adherence is judged from outputs, and output evidence is most fragile on value conflicts, where what matters is not which value a model mentions but which one it is willing to sacrifice. We provide evidence that this arbitration can be read from activations in a structured margin readout. We introduce Constitutional Value Potentials (CVP). For each value we learn a scalar potential from the hidden state: an internal pressure to preserve that value, supervised not by the prompt but by an independent judge's verdict on which value the model's own response actually preserved. The signed difference of two potentials is a priority margin. A constitutional clause becomes the claim that a margin stays positive, and a single monitor score flags when it does not. The monitor predicts conflict violations with AUROC up to 0.95, beats a strong hidden-state probe, and generalizes to held-out synthetic conflicts across three Qwen2.5 scales. The signal appears as the answer begins, from the prompt tail and first response token. Read this early, the same signal reveals whether an adversarial priority hack has actually pushed the model toward a violation, rather than only whether the prompt looks adversarial. The same directions also support intervention tests: under selected steering settings, moving along a value direction shifts judged trade-offs in the intended direction. Together, these results suggest that some constitution-relevant priorities are accessible as activation-space margins, rather than only as output behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。