将大模型推理拆分为探索与执行两阶段,提升效率与准确性。
Explore-Execute Chain: Towards an Efficient Structured Reasoning Paradigm
- 探索阶段随机生成简明计划,执行阶段精确落实策略。
- 在AIME'24上用12.4k tokens达53.3%准确率,仅需其他方法的1/6令牌。
- 可轻量适配新领域,仅用3.5%训练成本提升医学任务准确率14.5%。
许多大模型在行动前会进行规划,但规划与执行常混杂在单一生成过程中,仅靠提示词控制或分置于不同组件。我们指出这两阶段需要不同计算特性:规划需多样性与广度,执行则需精准与策略忠实性。将其视为单一连续流程会浪费大量令牌于重复推导,并使测试时探索替代策略成本高昂。本文提出【探索-执行链(E²C)】,在单模型内结构分离两个阶段:随机【探索】阶段生成简洁高层计划,确定性【执行】阶段忠实落实该计划。通过因果SFT与强化学习训练,确保探索信息丰富、执行忠实于计划。当计划短而有力时,额外推理算力可集中于探索而非反复解码完整答案。在AIME'24数据集上,$K=32$时,E²C-ReAct Loop达到53.3%准确率,仅使用12.4k令牌,优于树状思维($N=32$: 50.0%, 71.3k)。该结构亦支持轻量领域适配:探索聚焦微调(EF-SFT)仅更新规划阶段,使用标准SFT 3.5%的令牌,医学基准准确率最高提升14.5%。
原文摘要 · Abstract (English)
Many LLMs plan before they act, yet planning and execution are often still entangled in one long generation trace, enforced only through prompts, or split across separate components. We argue that these two stages call for different computation: planning benefits from diversity and breadth, whereas execution demands precision and faithful adherence to a chosen strategy. Treating them as a single undifferentiated chain wastes tokens on routine derivation and makes it costly to explore alternative strategies at test time. We present the \textbf{Explore-Execute Chain (E\textsuperscript{2}C)}, which keeps both stages in one model but separates them structurally: a stochastic \textit{Exploration} phase drafts a concise high-level plan, and a deterministic \textit{Execution} phase carries it out. Causal SFT and RL train this split so that exploration stays informative and execution remains plan-faithful. Once plans are short yet decisive, extra inference compute can be directed to exploration rather than to repeatedly decoding full solutions. On AIME'2024 at $K{=}32$, \textbf{E\textsuperscript{2}C-ReAct Loop} reaches 53.3\% accuracy with only 12.4k tokens, outperforming Tree-of-Thoughts ($N{=}32$: 50.0\%, 71.3k). The same structure also supports lightweight domain adaptation: \textbf{Exploration-Focused SFT (EF-SFT)} updates only the planning phase, uses 3.5\% of the tokens required by standard SFT, and improves medical benchmark accuracy by up to 14.5\%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。