让大模型学会自问自答中间问题，提升复杂任务解题能力
Asking the Right Questions: Improving Reasoning with Generated Stepping Stones
- 引入问题生成器，让模型主动构造解题中间步骤
- 生成的问题能显著提升不同能力模型的解题成功率
- 通过合成数据微调，可有效优化问题生成质量
近年来，大语言模型在数学和编程等复杂推理任务上取得显著进展。当面对单次无法解决的难题时，模型构建中间步骤（如简化、改写或子问题）的能力尤为重要。本文提出ARQ框架，在默认推理流程中加入问题生成模块，研究现代推理型LLM中中间步骤的特性与价值。实验表明，高质量的中间问题存在且可迁移，能显著帮助不同能力的模型完成目标任务。进一步将中间步骤生成视为后训练任务，通过SFT和强化学习在合成数据上微调，可有效提升模型生成有用中间问题的能力。
原文摘要 · Abstract (English)
Recent years have witnessed tremendous progress in enabling LLMs to solve complex reasoning tasks such as math and coding. As we start to apply LLMs to harder tasks that they may not be able to solve in one shot, it is worth paying attention to their ability to construct intermediate stepping stones that prepare them to better solve the tasks. Examples of stepping stones include simplifications, alternative framings, or subproblems. We study properties and benefits of stepping stones in the context of modern reasoning LLMs via ARQ (Asking the Right Questions), a simple framework that introduces a question generator to the default reasoning pipeline. We first show that good stepping stone questions exist and are transferrable, meaning that good questions can be generated, and they substantially help LLMs of various capabilities in solving the target tasks. We next frame stepping stone generation as a post-training task and show that we can fine-tune LLMs to generate more useful stepping stones by SFT and RL on synthetic data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。