用行动数据训练小模型,低成本检测智能体的隐秘操纵行为。
Training Deliberative Monitors for Black-Box Scheming Detection

- 仅通过观察动作序列,训练小型模型识别潜在阴谋行为。
- 在多个测试集上表现优于低成本大模型提示监控,且推理成本更低。
- 适合需要高性价比、低误报率的智能体安全监控场景。
随着自主智能体执行现实任务的能力增强,区分阴谋行为与正常任务追求可能成为核心的AI控制难题。现有监控方法通常依赖思维链或内部激活信息,或使用提示式前沿模型,但在部署中可能不可用、不可靠或成本高昂。本文研究仅基于动作的决策型监控:训练更小的开源权重模型,仅通过智能体轨迹检测阴谋与破坏行为,无需访问其推理过程或模型内部。该方法受决策对齐启发,利用阴谋规范从前沿教师模型中提取结构化推理,经独立裁判过滤后,通过监督微调与强化学习将高质量推理提炼为开源监控模型。我们在五个数据集上训练,并在六个分布外智能体不对齐基准上评估。结果表明,将本方法应用于 Qwen3.5-27B 模型,性能超过所有低成本提示前沿模型(Gemini 3.1 Flash-Lite、GPT-5.4 Nano、Claude Haiku 4.5),也优于 Gemini 2.5 Pro,且边际推理成本更低(每1000次评估的按令牌计费美元更低)。更强的提示式前沿模型(Gemini 3.1 Pro、GPT-5.4、Claude Sonnet 4.6、Claude Opus 4.6)虽性能更高,但边际推理成本高出约16–34倍。部分训练出的监控模型位于我们评估的所有监控方案的实证成本-性能帕累托前沿,为低成本、低误报率监控提供了实用替代方案。
原文摘要 · Abstract (English)
As autonomous agents become more capable of performing real-world tasks, distinguishing scheming behavior from benign task pursuit may become a central AI control problem. Existing monitors often rely on chain-of-thought access or internal activations, or use prompted frontier models, all of which can be unavailable, unreliable or expensive in deployment. In this work, we study action-only deliberative monitors: smaller open-weight models trained to detect scheming and sabotage from agentic trajectories without accessing the monitored agent's reasoning or model internals. Our method, inspired by deliberative alignment, uses a scheming specification to elicit structured rationales from a frontier teacher, filters them with a separate judge, and distills the highest-quality rationales into open-weight monitors with supervised fine-tuning and reinforcement learning. We train on five datasets, and evaluate across six out-of-distribution agentic misalignment benchmarks. We show that applying our method to Qwen3.5-27B yields higher performance than all low-cost frontier models as prompted monitors (Gemini 3.1 Flash-Lite, GPT-5.4 Nano, and Claude Haiku 4.5) and than Gemini 2.5 Pro, while also achieving lower marginal inference cost (token-metered USD per 1,000 evaluations). Stronger prompted frontier monitors (Gemini 3.1 Pro, GPT-5.4, Claude Sonnet 4.6, and Claude Opus 4.6) achieve higher performance but at roughly $16$--$34\times$ higher marginal inference cost. Several of our trained monitors are positioned on the empirical cost--performance Pareto frontier among the monitors we evaluate, providing practical low-cost, low-FPR alternatives to prompted frontier models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。