通过识别最早错误步骤,提升小模型推理的因果准确性。
CausalOPD: First-Wrong-Step Supervision for Distilling Causal Chain Reasoning

- 教师定位学生推理中首个违反规则的步骤,精准修正错误链。
- 相比传统方法,路径正确率提升23.4个百分点,错误归因率降至4.4%。
- 适合医疗、法律等需严格因果推理的高风险场景使用。
许多关键推理任务,如临床诊断、法律判断和工业故障分析,依赖于步骤相关的因果链,早期错误会持续传播,而正确结论可能掩盖无效推理过程。尽管大语言模型表现良好,但隐私、延迟和可控性促使将模型蒸馏为可本地部署的小模型。标准轨迹模仿无法在学生自身推理分布上纠正过程错误。本文提出CausalOPD,一种课程化在线过程蒸馏框架。知识增强型教师首先生成基于领域特定因果规则、实体关系与结构约束的推理轨迹。学生生成在线轨迹后,教师识别首个被验证违反约束的错误步骤(即最早不合规转移)。从该验证前缀开始,短程强化学习修复局部失败。因果阶段课程按证据级、机制级、结论级错误的传播顺序逐步推进。在三个领域中,CausalOPD相较于序列级在线过程蒸馏,平均路径正确率提升23.4个百分点,正确标签但错误推理率从15.7%降至4.4%。领域专用的80亿参数学生模型在所有领域均优于两个评估的专有参考模型。
原文摘要 · Abstract (English)
Many critical reasoning tasks, including clinical diagnosis, legal judgment, and industrial fault diagnosis, require step-dependent causal chains in which early errors propagate and correct conclusions can mask invalid reasoning. Although large language models perform well on such tasks, privacy, latency, and controllability motivate distillation into locally deployable models. Standard trajectory imitation does not correct process errors on the student's own rollout distribution. We propose CausalOPD, a curriculum online process distillation framework. A knowledge-augmented teacher first provides trajectories grounded in domain-specific causal rules, entity relations, and structural constraints. The student then generates on-policy trajectories, and the teacher identifies the first wrong step, defined as the earliest transition that verifiably violates available constraints. Starting from the verified prefix, short-horizon reinforcement learning repairs this localized failure. A causal-stage curriculum advances from evidence-level to mechanism-level and conclusion-level errors, following their propagation order. Across three domains, CausalOPD improves average path correctness by 23.4 percentage points over sequence-level online process distillation and reduces the right-label-wrong-reasoning rate from 15.7% to 4.4%. The domain-specific 8B students also surpass both evaluated proprietary references in path correctness across all domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。