arXiv:2510.01123cs.LGcs.AI2025-10被引 17

用思维迭代提升模型性能,同时降低推理成本。

Rethinking Thinking Tokens: LLMs as Improvement Operators

  • 设计并行-提炼-精炼框架,动态控制上下文长度
  • 在数学任务上实现比长链思维更高准确率且更低延迟
  • 适合追求高效高精度推理的开发者和研究者

推理训练促使大模型生成长链条思维(long CoT),虽提升准确率但增加上下文长度、计算开销与响应延迟。本文提出新思路:让模型利用元认知能力,在帕累托前沿探索更优组合。提出并行-提炼-精炼(PDR)框架:并行生成多样草稿,提炼至有限文本空间,再基于该空间精炼输出,可循环迭代。通过调节并行度,灵活控制上下文长度与计算成本,不再与生成总token数绑定。实验表明,使用当前模型实现的PDR在数学任务上优于长CoT,且延迟更低。当并行度为1时,演化出顺序精炼(SR)方法,性能超越长CoT。进一步以强化学习训练8B规模思维模型,使其适配PDR策略。在含可验证答案的数学任务中,迭代式推理在匹配串行预算下超越单次推理基线,其中PDR提升最显著(如AIME 2024 +11%,AIME 2025 +9%)。

原文摘要 · Abstract (English)

Reasoning training incentivizes LLMs to produce long chains of thought (long CoT), which among other things, allows them to explore solution strategies with self-checking. This results in higher accuracy, but inflates context length, token/compute cost, and answer latency. We ask: Can current models leverage their metacognition to provide other combinations on this Pareto frontier, e.g., better accuracy with lower context length and/or latency? Abstractly, we view the model as an improvement operator on its own "thoughts" with a continuum of possible strategies. We identify an interesting inference family Parallel-Distill-Refine (PDR), which performs the following: (i) generate diverse drafts in parallel; (ii) distill them into a bounded, textual workspace; and (iii) refine conditioned on this workspace, producing an output that seeds the next round. Importantly, context length (hence compute cost) is controllable via degree of parallelism, and is no longer conflated with the total number of generated tokens. We report PDR instantiations of current models that give better accuracy than long CoT while incurring lower latency. Setting degree of parallelism to 1 yields an interesting subcase, Sequential Refinement (SR) (iteratively improve a single candidate answer) which provides performance superior to long CoT. Success of such model orchestrations raises the question whether further training could shift the Pareto frontier. To this end, we train an 8B thinking model with Reinforcement Learning (RL) to make it consistent with PDR as the inference method. On math tasks with verifiable answers, iterative pipelines surpass single-pass baselines at matched sequential budgets, with PDR delivering the largest gains (e.g., +11% on AIME 2024 and +9% on AIME 2025).

思维链推理优化模型调度强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。