让弱模型学会强模型的解题策略,而非死记步骤。
Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning

- 用可复用的解题策略替代具体步骤模仿,提升泛化能力。
- 在四个数学基准上平均得分比最强基线高2.2分。
- 适合想提升大模型推理能力的研究者和工程师。
将强语言模型的推理能力迁移到弱模型,通常依赖于对特定解题路径的模仿,仅传递‘答什么’而非‘怎么想’。这种路径级模仿易导致对实例细节的记忆,限制解决新问题的能力。我们提出策略引导的策略优化(SGPO),以可复用的策略提炼替代实例级路径模仿。SGPO从强模型响应中提取结构化策略描述,并为每道题构建自主路径与策略引导路径,实现有无策略指导下的行为对比。框架解决两个关键问题:如何蒸馏?采用令牌级前向KL目标,选择性地将策略条件引起的分布偏移传递给无指导策略,同时通过近端约束保证稳定性。何时蒸馏?自适应实例级加权机制在自主探索不足时增强指导,在模型能力提升时降低指导强度。在两个模型家族的四个数学基准上的实验表明,SGPO持续优于SFT、在线强化学习及混合策略基线,在Qwen2.5-7B-Instruct上平均分高出最强基线2.2分。分析显示,前向KL提供内在选择性的蒸馏信号,优于直接路径模仿;策略蒸馏与基础模型能力呈互补缩放关系。
原文摘要 · Abstract (English)
Distilling reasoning capabilities from strong to weak language models typically involves imitating specific solution trajectories, effectively transferring what to answer rather than how to reason. This trajectory-level imitation encourages memorization of instance-specific steps rather than acquisition of transferable problem-solving skills, limiting generalization to novel problems. We propose Strategy-Guided Policy Optimization (SGPO), which replaces instance-level trajectory imitation with reusable strategy distillation. SGPO extracts structured strategy descriptions from strong-model responses and, for each problem, constructs both autonomous and strategy-guided trajectories to enable direct comparison of the model's behavior with and without strategic guidance. The framework then addresses two key questions. For how to distill, a token-level forward-KL objective selectively transfers the distributional shift induced by strategy conditioning into the unguided policy, with proximal constraints ensuring stability. For when to distill, adaptive instance-level weighting strengthens guidance when autonomous exploration falls short and reduces it as the model's own competence grows. Experiments on four mathematical benchmarks across two model families show that SGPO consistently outperforms SFT, on-policy RL, and hybrid-policy baselines, improving the average score by 2.2 points over the strongest baseline on Qwen2.5-7B-Instruct. Analysis reveals that the forward-KL objective provides an inherently selective distillation signal that outperforms direct trajectory imitation, and that strategy distillation exhibits complementary scaling with base model capability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。