让大模型在推理时提前发出行为提示，提升安全监控效率
Behavior Cue Reasoning: Monitorable Reasoning Improves Efficiency and Safety through Oversight
- 训练模型在特定行为前输出特殊标记序列，作为可监控信号
- 能减少50%冗余推理token，使安全动作恢复率从46%提升至96%
- 无需牺牲性能，适合需要高安全性的复杂任务场景
大型语言模型的推理过程难以被有效监督，许多行为偏差直到推理结束才显现。为此，我们提出行为提示推理（Behavior Cue Reasoning），通过训练模型在特定隐性或显性行为前生成特殊标记序列，作为兼具信号与控制功能的双重机制。在使用强化学习微调外部监测器时，仅依赖行为提示所传递的信息即可使监测器在复杂数学问题求解中裁剪掉高达50%的冗余推理令牌。当由近似最优的规则监测器使用时，在过度违规会导致失败的环境中，该方法使原本会生成不安全动作的80%推理轨迹得以恢复安全行为，成功率从46%提升至96%以上。跨两个模型族和三个领域的评估表明，该方法在不损害性能的前提下显著提升了推理的可监控性与可控性。本工作推动了可扩展监督的发展，展示了被监控模型自身可通过训练实现更利于监督的推理方式。
原文摘要 · Abstract (English)
Reasoning in Large Language Models (LLMs) poses a challenge for oversight as many misaligned behaviors do not surface until reasoning concludes. To address this, we introduce Behavior Cue Reasoning for making LLM reasoning more controllable and monitorable. Behavior Cues are special token sequences that a model is trained to emit immediately before specific implicit and explicit behaviors, acting as dual purpose signal and control levers. When fine-tuning a weaker external monitor with Reinforcement Learning for reasoning oversight, a compressed view of only information surfaced by Behavior Cues is sufficient signal for the monitor to prune up to 50% of otherwise wasted reasoning tokens in complex math problem solving. When leveraged by an almost optimal rule-based monitor in an environment where excessive constraint violations results in failure, Behavior Cues allows for the recovery of safe actions from 80% of reasoning traces that would otherwise end with the proposal of an unsafe action, more than doubling the success rate from 46% to 96%. Through evaluation across two model families and three domains, we show that Behavior Cue Reasoning improves reasoning monitorability and controllability with no cost to performance. More broadly, our work progresses scalable oversight by demonstrating how the monitored model itself can be trained to reason more tractably to oversight. Code: https://github.com/christopherzc/behavior-cues
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。