通过精准修正数据提升大模型推理能力，同时避免遗忘旧知识。
Surgical Post-Training: Proximal On-Policy Distillation for Reasoning with Knowledge Retention
- 用智能修正生成贴近原数据的训练样本，提升学习效率。
- 仅用4000组数学题对，使模型准确率平均提升6.2%。
- 适合需要高效微调且保留已有知识的场景。
通过后训练向大语言模型注入新推理知识常导致灾难性遗忘。尽管近期研究强调了在线策略数据的重要性，但认为KL散度无法缓解遗忘。与此相反，我们从分析和实证两方面证明，KL约束的奖励机制在后训练中对知识保留起关键作用。由此提出外科后训练（SPOT），一种近端在线策略蒸馏框架，可高效优化推理并保持原有知识。SPOT包含：(1) 采用专家模型进行精准修正的数据校正流程，通过最小修改生成近端在线策略数据；(2) 基于奖励的二元交叉熵目标，有效增强推理并减轻遗忘。实验表明，仅使用4000组校正后的数学对，SPOT在Qwen3-8B上使域内与域外任务平均准确率提升6.2%，在8×H800 GPU上仅需16分钟训练。此外，SPOT为后续强化学习提供更优初始化，显著提升性能上限。
原文摘要 · Abstract (English)
Injecting new reasoning knowledge into Large Language Models (LLMs) via post-training often induces catastrophic forgetting. Recent studies emphasize the importance of on-policy data but suggest that KL-divergence fails to mitigate forgetting. In contrast, we show, both analytically and empirically, that the KL-constrained reward formulation actually plays a critical role in retaining knowledge during post-training. This motivates our Surgical Post-Training (SPOT), a proximal on-policy distillation framework designed to optimize reasoning efficiently while preserving prior knowledge. SPOT consists of (1) a data rectification pipeline employing an Oracle to surgically correct erroneous steps via minimal edits, generating proximal on-policy data; and (2) a reward-based binary cross-entropy objective essential for enhancing reasoning and mitigating forgetting. Empirically, with only 4k rectified math pairs, SPOT improves Qwen3-8B's accuracy by 6.2% on average across in-domain and out-of-domain tasks, requiring merely 16-minute model training on 8x H800 GPUs. Moreover, SPOT provides a superior initialization for subsequent reinforcement learning, significantly elevating the performance ceiling. Code: https://github.com/Visual-AI/SPoT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。