让大模型学会正确推理因果,而非仅猜对答案。
Epistemic Regret Minimization: Label-Free Causal Critique Beyond Outcome Reward
- 通过分析模型推理过程中的因果结构缺陷来纠错
- 在无答案标签时仍能提升模型表现,准确率提高53-59个百分点
- 适合需要严谨因果推理的AI安全与可信系统研究者
大型语言模型虽能正确回答因果问题,却常因错误理由。现有强化学习方法只奖励结果,忽视推理原因,导致模型固化相关性捷径,我们称之为‘奖励固化’。本文提出‘认知后悔最小化’(Epistemic Regret Minimization, ERM)框架,不依赖真实因果图或正确答案,而是从推理痕迹中识别未检验混杂因素、相关与干预混淆、未封闭后门路径等因果缺陷。该框架可实现无标签操作,在单次任务中检测并修复因果错误,跨任务积累干预证据形成奖励信号。六种前沿大模型在1,360个场景测试中,重推理模型(如GPT-4 Turbo、GPT-5.2)仅靠结果修正恢复率25–31%,而采用因果批判后恢复率达78–91%,提升53–59个百分点。标准测试时方法(自一致性、Best-of-N、Self-Refine)在因果任务上反而劣于仅基于结果的重提示;而ERM将残余层级坍缩从55–70%降至4%。理论证明仅基于结果的奖励无法弥补此差距,模拟验证表明认知反馈可使性能超越结果基线38倍。
原文摘要 · Abstract (English)
Large language models can answer causal questions correctly for the wrong reasons. Current RL methods reward \emph{what} a model concludes but ignore \emph{why}, reinforcing correlational shortcuts -- a failure we call \emph{Reward Entrenchment}. We introduce \emph{Epistemic Regret Minimization} (\erm), a framework that critiques the causal \emph{structure} of a model's reasoning trace rather than its answer. Applying established causal principles, \erm flags unexamined confounders, correlation--intervention conflation, and unchecked back-door paths from exposed reasoning traces. The framework admits \emph{label-free} operation -- without the true causal graph or correct answer -- and we separately distinguish favorable benchmark-derived critique, error-direction cues, and fully label-free judge-generated critique in the experiments. Within a single episode, \erm detects and repairs causal reasoning errors; across episodes, it accumulates interventional evidence into a reward signal applicable where no answer key exists. Experiments on 1,360 scenarios across six frontier LLMs show that reasoning-heavy models (GPT-4 Turbo, GPT-5.2) resist outcome-only correction (25--31\% recovery) yet respond to causal critique (78--91\%), gaining $+53$--$59$ pp. Standard test-time methods (self-consistency, Best-of-$N$, Self-Refine) \emph{underperform} outcome-only reprompting on causal tasks, while ERM reduces residual Rung Collapse from 55--70\% to 4\%. A separation theorem proves outcome-only reward cannot close this gap; a controlled simulation confirms epistemic feedback does, outperforming outcome-only baselines 38-fold.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。