通过引导式蒸馏让模型学会吸收批评,提升推理能力。
Critique-Guided Distillation for Robust Reasoning via Refinement
- 将批评作为训练时监督信号,不参与推理
- 数学推理基准上平均提升7%,最高达15%
- 保留指令遵循能力,适合需要可靠推理的场景
监督微调常导致模型模仿输出而非内化推理过程。尽管基于批评的方法有潜力,但直接训练模型生成批评(如CFT)会导致输出格式漂移和泛化能力下降。我们提出批判引导蒸馏(CGD),将批评生成与批评吸收解耦:学生模型在教师批评指导下修正错误响应,而批评仅在训练时使用,推理时消失。控制消融实验表明,推理性能提升源于教师反馈的准确性和相关性。在五个模型家族中,CGD持续优于CFT和标准蒸馏,在数学推理基准上平均提升7%,在AMC23上达+15.0%,MATH-500上达+12.2%。在AIME24和AIME25等难题上,CGD显著提高Pass@1,并在低Pass@k下表现更强,说明单样本推理质量提升。更重要的是,CGD保持了良好的指令遵循能力,而CFT在此项上下降21.3%(IFEval)。CGD是一种高效且无需推理时额外开销的推理导向训练范式。
原文摘要 · Abstract (English)
Supervised fine-tuning with expert demonstrations often produces models that imitate outputs without internalizing the reasoning processes needed for robust generalization. While critique-based approaches show promise, training models to generate critiques directly, such as Critique Fine-Tuning (CFT), can lead to output-format drift and degradation of general capabilities. We propose Critique-Guided Distillation (CGD), a training framework that decouples critique consumption from critique generation. During fine-tuning, the student is trained to refine flawed responses conditioned on teacher critiques. CGD treats critiques as a \textit{training-time-only} supervision signal, encouraging internalization of error-aware reasoning: critiques guide learning but are absent at inference. Controlled ablations confirm that these reasoning gains are directly driven by the specificity and relevance of the teacher's feedback. Across five model families, CGD consistently outperforms CFT and standard distillation on mathematical reasoning benchmarks, yielding 7\% average improvements and gains of up to +15.0\% on AMC23 and +12.2\% on MATH-500. On challenging competition problems such as AIME24 and AIME25, CGD achieves substantially higher Pass@1 and stronger performance at low Pass@k, indicating improved reasoning quality per sample. Importantly, CGD preserves general instruction-following capabilities where CFT degrades significantly ($-$21.3\% on IFEval). These results position CGD as a practical and compute-efficient intermediate training paradigm for reasoning-centric tasks without introducing architectural inference-time overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。