通过可读性与完整性评估模型推理过程的可监控性。
Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbosity
- 结合忠实性与冗长度,构建综合可监控性评分
- 部分模型看似忠实却因遗漏关键因素难以监控
- 适用于安全验证与推理模型评估的场景
链式思维(CoT)输出使我们能读取模型的逐步推理过程。由于任何长串推理都必须经过这一文本痕迹,其质量直接反映模型的内部思考。这种可见性有助于发现不安全或偏离对齐的行为(可监控性),但前提是CoT真实反映内部推理(忠实性)。完全测量忠实性困难,因此研究者常关注模型在添加提示后改变答案的情形。该代理方法仅捕捉部分不忠实现象,且在模型保持原答案时丢失信息,无法考察与提示无关的推理层面。本文通过引入冗长度——即CoT是否列出解决任务所需的所有因素——扩展了可监控性的评估。将忠实性与冗长度结合,形成统一的可监控性分数,衡量CoT作为模型外部“工作记忆”的表现,这正是依赖CoT监控的安全机制所依赖的核心属性。我们在BBH、GPQA和MMLU数据集上评估指令微调与推理模型。结果表明,某些模型虽表现忠实,却因遗漏关键因素而难以监控,且不同模型家族间的可监控性差异显著。我们已开源基于Inspect库的评估代码,支持未来可复现研究。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) outputs let us read a model's step-by-step reasoning. Since any long, serial reasoning process must pass through this textual trace, the quality of the CoT is a direct window into what the model is thinking. This visibility could help us spot unsafe or misaligned behavior (monitorability), but only if the CoT is transparent about its internal reasoning (faithfulness). Fully measuring faithfulness is difficult, so researchers often focus on examining the CoT in cases where the model changes its answer after adding a cue to the input. This proxy finds some instances of unfaithfulness but loses information when the model maintains its answer, and does not investigate aspects of reasoning not tied to the cue. We extend these results to a more holistic sense of monitorability by introducing verbosity: whether the CoT lists every factor needed to solve the task. We combine faithfulness and verbosity into a single monitorability score that shows how well the CoT serves as the model's external `working memory', a property that many safety schemes based on CoT monitoring depend on. We evaluate instruction-tuned and reasoning models on BBH, GPQA, and MMLU. Our results show that models can appear faithful yet remain hard to monitor when they leave out key factors, and that monitorability differs sharply across model families. We release our evaluation code using the Inspect library to support reproducible future work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。