用探针检测推理中的虚假步骤，让模型更真实地思考。
Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning
- 用注意力探针识别模型完成答案后添加的无效推理步骤。
- 在四个任务中减少11%到100%的虚假推理，提升真实推理比例24个百分点。
- 无需人工标注，适配不同模型，让推理过程更可信、更简洁。
推理模型常在内部已确定答案后，才生成看似合理的推理链，形成‘推理表演’——这些步骤对正确性无贡献，浪费计算资源且干扰可解释性。本文提出 ProFIL（Probe-Filtered RL），作为 Group Relative Policy Optimization (GRPO) 的即插即用扩展，通过一次训练的多头注意力探针，仅基于冻结基础模型的内部激活，识别出答案提交后的冗余推理步骤；在强化学习过程中，若某次采样的探针得分超过阈值，则其优势值置零。核心发现：一个基于验证器标签、无需人工标注的探针，在冻结基模型上训练后，能稳定抑制推理表演，且避免先前研究预测的强化学习混淆失效问题。在 GSM8K、LiveCodeBench、ToolUse、MMLU-Redux 四个推理领域，以及 Llama-8B、Qwen-7B 两种架构下，ProFIL 将事后推理（post-commitment theater）减少 11%–100%，忠实推理比例提升（如在 LiveCodeBench 上独立使用 Claude 3.7 Sonnet 判定时 +24 个百分点），链长缩短 4%–19%，同时保持或提升任务准确率。相比长度惩罚基线，性能提升源自语义承诺检测而非单纯压缩链长。探针权重、训练配置与采样数据已在所有四领域公开。
原文摘要 · Abstract (English)
Reasoning models post-hoc rationalize answers they have already committed to internally, producing chains of *reasoning theater*: deliberative-looking steps that contribute nothing to correctness. This wastes inference tokens, pollutes interpretability, and obscures what the model actually computed. We introduce **ProFIL** (**Pro**be-**Fil**tered Reinforcement Learning) to *reduce theater, increase chain-of-thought faithfulness, and shrink chain length* in a single, drop-in extension to Group Relative Policy Optimization (GRPO). A multi-head attention probe is trained *once* on the *frozen* base model to detect post-commitment steps from internal activations alone; during GRPO, rollouts whose probe score exceeds a threshold have their advantage zeroed. *Our central finding is that a probe trained on a frozen base, with verifier-derived labels and no human annotation, provides a stable signal that suppresses theater while resisting the RL-obfuscation failure mode predicted by prior work.* Across four reasoning domains (GSM8K, LiveCodeBench, ToolUse, MMLU-Redux) and two model architectures (Llama-8B, Qwen-7B), ProFIL reduces post-commitment theater by **11--100%**, raises faithful-fraction (e.g., +24pp on LiveCodeBench under an independent Claude 3.7 Sonnet judge), and shortens chains by 4--19%, all while preserving or improving task accuracy. ProFIL also beats a matched length-penalty GRPO baseline, isolating the gain as semantic commitment-detection rather than chain compression. Probe weights, training configurations, and rollouts are released across all four domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。