让AI解释时能被程序验证,但解释未必真实反映思考过程。
Decodable but Not Faithful: Coupling Natural-Language Rationales to Programmatic Verifiers

- 在推理中插入断言,用辅助头预测程序验证结果
- 生成的解释可被验证,但内容常与实际算法无关
- 适合研究AI可解释性与验证机制的学者
语言模型能生成看似合理的预测解释,但这些解释未必忠实反映其内部推理。我们提出验证器耦合推理框架,在推理轨迹中插入内联断言,并训练辅助一致性头,从解释片段的隐藏状态预测程序验证输出。核心发现是:可解码性与真实性之间存在差距——一致性训练可使验证信息从解释表示中被可靠解码,但可解码不等于真实。在LeanCheck(形式化证明)中,仅用解释或仅用证明的池化方法在反事实冲突下实现完美方向分离;在KataGo(围棋引擎)中,评论片段以81%准确率编码10类胜率区间;在代码任务中,模型达到98.6%的耦合度,但生成解释虽流畅且结构正确,却描述无关算法。预训练与从零训练对比显示该差距非容量所致。合成激活修补证实因果影响(73-89%对比31%基线),FEVER实验表明仅用证据池化能隔离真实证据敏感性,但牺牲原始准确率,逐断言分析显示一致性损失更偏利于细粒度断言而非二元断言。结果表明,一致性损失是有效的诊断工具与表征塑造手段,但不足以保证真实推理。
原文摘要 · Abstract (English)
Language models can generate plausible rationales for their predictions, but these explanations may not faithfully represent the model's internal reasoning. We propose verifier-coupled reasoning, a framework that inserts inline claims into reasoning traces and trains an auxiliary consistency head to predict programmatic verifier outputs from rationale-span hidden states. The central finding is a gap between decodability and faithfulness: consistency training reliably makes verifier information decodable from rationale representations, but decodability does not guarantee faithful generation. In LeanCheck (formal theorem proving), rationale-only and proof-only pooling achieve perfect directional separation under counterfactual conflict. In KataGo (Go engine), commentary spans encode 10-way win-rate buckets at 81% accuracy. Yet in a code setting, the model achieves 98.6% coupling while its generated explanations remain unfaithful: fluent prose with correct structured claims, but describing unrelated algorithms; a controlled pretrained-vs-from-scratch comparison shows the gap is not capacity-driven. Synthetic activation patching confirms causal influence (73-89% vs. 31% baseline), FEVER reveals that evidence-only pooling isolates genuine evidence sensitivity at the cost of raw accuracy, and per-claim analysis shows that consistency loss disproportionately benefits fine-grained claims over binary ones. These results establish that consistency losses are effective diagnostics and representation-shaping tools, but not sufficient conditions for faithful reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。