让大模型推理更可信，防止用假理由骗过评分器。
Causally-Enhanced Reinforcement Policy Optimization
- 用可微分的因果一致性信号增强奖励函数，追踪从提示到答案的生成路径。
- 在4个数据集上平均提升准确率5.49%（最高9.58%），且对虚假推理更鲁棒。
- 无需修改模型结构，适合提升需要可靠逻辑的大模型应用。
以强化学习目标训练的大语言模型常通过捷径策略获得表面正确的答案，即输出正确但推理过程虚假或不一致，对微小因果扰动敏感。本文提出因果增强型策略优化（CE-PO），一种可直接集成的奖励重构框架，通过雅可比矩阵估算模型内部影响，反事实硬化信号以抑制无关线索，并将因果一致性得分与任务准确率反馈通过Minkowski组合器融合，实现精度与一致性之间的单一可调权衡。该统一奖励可无缝接入PPO/GRPO算法，无需架构修改。在多个推理基准和因果压力测试中，CE-PO有效降低奖励劫持和非忠实思维链现象，提升对相关性-因果性混淆及轻量反事实编辑的鲁棒性，同时保持近似基线的准确率。跨4个数据集的实验表明，CE-PO平均提升准确率5.49%（最高达9.58%），并显著增强对因果关系扰动的稳定性。
原文摘要 · Abstract (English)
Large language models (LLMs) trained with reinforcement objectives often achieve superficially correct answers via shortcut strategies, pairing correct outputs with spurious or unfaithful reasoning and degrading under small causal perturbations. We introduce Causally-Enhanced Policy Optimization (CE-PO), a drop-in reward-shaping framework that augments policy optimization with a differentiable proxy for causal coherence along the generation pathway from prompt (Z) to rationale (X) to answer (Y). CE-PO estimates model-internal influence with Jacobian-based sensitivities, counterfactually hardens these signals to suppress nuisance cues, and fuses the resulting coherence score with task-accuracy feedback via a Minkowski (power-mean) combiner, exposing a single tunable between accuracy and coherence trade-off. The unified reward integrates with PPO/GRPO without architectural changes. Across reasoning benchmarks and causal stress tests, CE-PO reduces reward hacking and unfaithful chain-of-thought while improving robustness to correlation-causation flips and light counterfactual edits, all at near-parity accuracy. Experimental results across 4 datasets show that CE-PO improves accuracy over baselines by 5.49% on average (up to 9.58%), while improving robustness to correlation-causation flips and light counterfactual edits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。