让大模型学会用推理流程，训练时自蒸馏提升推理能力
Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoning
- 用增强版模型作为教师，对原模型进行自蒸馏训练
- 在数学和文本分类任务上，性能比基线高10.83%
- 推理流程可作临时训练工具，不需永久保留
推理阶段的流程框架能显著提升大语言模型在复杂推理任务上的表现。然而，这种外部流程不会改变模型本身的内在能力。为此，我们提出「在策略流程自蒸馏」（OPHSD），利用带有流程增强的当前模型作为教师，对自身进行自蒸馏，从而引入超越训练数据的额外监督信号。该方法将特定任务的流程能力内化到学生模型中，实现跨多种推理任务的强泛化性和独立运行性能。在文本分类的草稿-验证流程和数学推理的计划-求解流程上，OPHSD始终优于强基线（如在HMMT25上比OPSD高出10.83%）。进一步分析表明，推理时重新接入流程并无额外收益，甚至可能降低性能，说明复杂流程可作为临时训练支架，其优势可永久融入基础模型。
原文摘要 · Abstract (English)
Inference-time harnesses substantially improve large language models on complex reasoning tasks. However, the intrinsic capabilities of the underlying model remain unchanged by the addition of these external workflows. To bridge this gap, we introduce \emph{On-Policy Harness Self-Distillation} (OPHSD), which employs the harness-augmented current model as a teacher for self-distillation, thereby introducing extra supervisory signals from the harness beyond training data. OPHSD internalizes task-specific harness capabilities into the student model, yielding robust generalizability and strong standalone performance across diverse reasoning tasks. Evaluated across draft--verify harness for text classification and plan--solve for mathematical reasoning tasks, OPHSD consistently outperforms strong baselines (e.g., +10.83\% over OPSD on HMMT25). Our analysis further indicates that reattaching the harness during inference yields no additional benefits and can even degrade performance, suggesting that complex harnesses need not always be permanent fixtures; instead, they can serve as temporary training scaffolds whose benefits are permanently fed back into the base model. Our code and training data are available at https://github.com/zzy1127/OPHSD-On-Policy-Harness-Self-Distillation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。