用反事实模拟训练提升大模型推理过程的可信度
Counterfactual Simulation Training for Chain-of-Thought Faithfulness
- 通过反事实输入测试推理一致性，奖励能准确预测输出的思维链
- 在2350亿参数模型上，监控准确率提升35点，可模拟性提高2点
- 适合需要验证模型推理可靠性的研究人员和安全评估团队
检查思维链（Chain-of-Thought, CoT）是理解大语言模型输出原因的常用方法，但其可信度问题严重限制了分析价值。本文提出反事实模拟训练（Counterfactual Simulation Training, CST），通过奖励那些使模拟器能在反事实输入下准确预测模型输出的思维链，以提升CoT的可信度。CST应用于两种场景：(1) 基于提示词的反事实监测，识别模型对无关特征、奖励作弊或顺从行为的依赖；(2) 面向通用模型生成的反事实模拟，促进更忠实、可泛化的推理。实验表明，在高达2350亿参数的模型上，CST显著提升了基于提示词反事实的监控准确率（+35点）以及通用反事实下的可模拟性（+2点）。结果还显示：(1) CST优于提示基线；(2) 使用LLM重写不可信思维链效率比纯强化学习高5倍；(3) 可信度改进无法推广至抑制性提示（而非诱导性提示）；(4) 大模型本身不具天然更高可信度，但更受益于CST。这些结果表明CST可普遍提升思维链可信度，具有广阔的应用前景。
原文摘要 · Abstract (English)
Inspecting Chain-of-Thought reasoning is among the most common means of understanding why an LLM produced its output. But well-known problems with CoT faithfulness severely limit what insights can be gained from this practice. In this paper, we introduce a training method called Counterfactual Simulation Training (CST), which aims to improve CoT faithfulness by rewarding CoTs that enable a simulator to accurately predict a model's outputs over counterfactual inputs. We apply CST in two settings: (1) CoT monitoring with cue-based counterfactuals, to detect when models rely on spurious features, reward hack, or are sycophantic, and (2) counterfactual simulation over generic model-based counterfactuals, to encourage models to produce more faithful, generalizable reasoning in the CoT. Experiments with models up to 235B parameters show that CST can substantially improve monitor accuracy on cue-based counterfactuals (by 35 accuracy points) as well as simulatability over generic counterfactuals (by 2 points). We further show that: (1) CST outperforms prompting baselines, (2) rewriting unfaithful CoTs with an LLM is 5x more efficient than RL alone, (3) faithfulness improvements do not generalize to dissuading cues (as opposed to persuading cues), and (4) larger models do not show more faithful CoT out of the box, but they do benefit more from CST. These results suggest that CST can improve CoT faithfulness in general, with promising applications for CoT monitoring. Code for experiments in this paper is available at https://github.com/peterbhase/counterfactual-simulation-training
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。