研究训练方式如何影响模型推理过程的可监控性。
Reasoning Under Pressure: How do Training Incentives Influence Chain-of-Thought Monitorability?
- 通过预测隐变量评估推理过程可监控性
- 对抗性优化会降低监控效果，但直接优化监控性无效
- 揭示训练激励对安全监控的关键影响
能够输出自然语言推理链（CoT）的AI系统为安全监控提供了可能——我们可以监控其推理过程，识别有害意图。然而，推理链是否真实反映底层思维，以及能否被有效监控，可能受训练方式影响。本文研究不同训练激励对模型可监控性的影响。提出一种新方法：通过监控器能否根据推理内容预测关键隐变量来衡量可监控性。在控制准确率的前提下，未发现长度惩罚和KL正则化有显著影响；但对抗性优化（惩罚监控器准确率）会损害监控性能；而直接优化监控性也未能带来可靠提升。代码已公开于 https://github.com/QiyaoWei/reasoning-under-pressure。
原文摘要 · Abstract (English)
AI systems that output their reasoning in natural language offer an opportunity for safety -- we can \emph{monitor} their chain of thought (CoT) for undesirable reasoning, such as the pursuit of harmful objectives. However, the extent to which CoT faithfully reflects the underlying reasoning process, and hence the extent to which it can be usefully monitored, may be influenced by certain aspects of training. We investigate how different \emph{training incentives}, applied to a reasoning model, affect its monitorability. We introduce a novel methodology for measuring monitorability according to whether a monitor can predict a key latent variable using the model's reasoning. When controlling for accuracy, we do not find evidence for consistent effects from commonly used incentives (length penalties and KL regularisation), but we find that adversarial optimisation (penalising monitor accuracy) degrades monitor performance, while direct optimisation for monitorability does not reliably lead to improvements. Our code is available at https://github.com/QiyaoWei/reasoning-under-pressure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。