通过分轮结构化分解,让大模型推理更快更可控。
Done Is Better than Perfect: Unlocking Efficient Reasoning by Structured Multi-Turn Decomposition
- 将思维链拆分为多轮交互,每轮聚焦一个思考单元
- 在MATH数据集上实现70%的输出令牌与首字延迟降低
- 支持用户中途终止或继续,适合对效率敏感的应用
大型推理模型因过长的思维链导致首令牌延迟和整体延迟过高。传统思维链混合多个思考单元,难以管理。本文提出多轮分解(MinD),将思维链转化为显式、结构化、分轮进行的交互序列,每轮包含一个思考单元并生成对应答案,后续轮次可反思、验证、修正或探索替代方案。该方法不仅加快答案生成,还支持用户在任意轮次中断或继续。采用监督微调后强化学习的范式训练,先用另一大模型将原始输出重构为多轮格式,再微调模型。发现微调后模型反而使用更多令牌,故引入GRPO等强化学习算法,优先选择少轮次且正确的输出。在R1-Distill模型和MATH数据集上训练,MinD可实现约70%的输出令牌数与首令牌时间(TTFT)减少,同时在MATH-500、AIME24、AMC23和GPQA-Diamond等推理基准上保持竞争力。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) are criticized for the excessively lengthy Chain-of-Thought (CoT) to derive the final answer, suffering from high first-token and overall latency. Typically, the CoT of LRMs mixes multiple thinking units; each unit attempts to produce a candidate answer to the original query. Hence, a natural idea to improve efficiency is to reduce the unit number. Yet, the fact that the thinking units in vanilla CoT cannot be explicitly managed renders doing so challenging. This paper introduces Multi-Turn Decomposition (MinD) to decode conventional CoT into a sequence of explicit, structured, and turn-wise interactions to bridge the gap. In MinD, the model provides a multi-turn response to the query, where each turn embraces a thinking unit and yields a corresponding answer. The subsequent turns can reflect, verify, revise, or explore alternative approaches to both the thinking and answer parts of earlier ones. This not only makes the answer delivered more swiftly, but also enables explicit controls over the iterative reasoning process (i.e., users may halt or continue at any turn). We follow a supervised fine-tuning (SFT) then reinforcement learning (RL) paradigm to realize MinD. We first rephrase the outputs of an LRM into multi-turn formats by prompting another LLM, and then tune the LRM with such data. Observing that the tuned model tends to consume even more tokens than the original one (probably due to that the multi-turn formats introduce additional answer tokens), we advocate leveraging RL algorithms like GRPO to prioritize correct outputs with fewer turns. Trained on the MATH dataset using R1-Distill models, MinD can achieve up to ~70% reduction in both output token usage and time to first token (TTFT), while maintaining competitive performance on reasoning benchmarks such as MATH-500, AIME24, AMC23, and GPQA-Diamond.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。