让小模型学会分步解题，提升复杂问题求解能力。
PLAN-TUNING: Post-Training Language Models to Learn Step-by-Step Planning for Complex Problem Solving
- 从大模型提取分解步骤，指导小模型学习规划过程。
- 在GSM8K和MATH上性能比基线平均提升7%。
- 在奥数和竞赛题上泛化能力更强，提升超10%。
近期，将复杂问题分解为简单子任务——类人自然规划的关键步骤——显著提升了大语言模型（LLMs）的性能。然而，在后训练阶段利用此类规划结构来增强小型开源LLMs的表现仍研究不足。为此，我们提出PLAN-TUNING，一种统一的后训练框架：(i) 从大规模LLMs中提炼合成的任务分解路径（称为“规划轨迹”），(ii) 通过监督与强化学习目标微调小型模型，使其模仿这些规划过程以提升复杂推理能力。在GSM8K和MATH基准上，计划微调模型平均优于强基线约7%。此外，计划微调模型在跨领域数据集上表现出更优泛化能力，分别在OlympiadBench和AIME 2024上实现平均约10%和12%的性能提升。详细分析表明，规划轨迹有效增强了复杂推理能力，证明了PLAN-TUNING是提升小型LLMs特定任务表现的有效策略。
原文摘要 · Abstract (English)
Recently, decomposing complex problems into simple subtasks--a crucial part of human-like natural planning--to solve the given problem has significantly boosted the performance of large language models (LLMs). However, leveraging such planning structures during post-training to boost the performance of smaller open-source LLMs remains underexplored. Motivated by this, we introduce PLAN-TUNING, a unified post-training framework that (i) distills synthetic task decompositions (termed "planning trajectories") from large-scale LLMs and (ii) fine-tunes smaller models via supervised and reinforcement-learning objectives designed to mimic these planning processes to improve complex reasoning. On GSM8k and the MATH benchmarks, plan-tuned models outperform strong baselines by an average $\sim7\%$. Furthermore, plan-tuned models show better generalization capabilities on out-of-domain datasets, with average $\sim10\%$ and $\sim12\%$ performance improvements on OlympiadBench and AIME 2024, respectively. Our detailed analysis demonstrates how planning trajectories improves complex reasoning capabilities, showing that PLAN-TUNING is an effective strategy for improving task-specific performance of smaller LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。