arXiv:2509.24238cs.AIcs.CL2025-09被引 4

让大模型按题难易自动调节推理深度,省算力提准确率。

Learning to Ponder: Adaptive Reasoning in Latent Space

  • 用潜空间控制器动态决定是否继续推理,无需改主干模型。
  • 在GSM8K和MATH500上以更低算力实现更高准确率。
  • 适合需要高效推理的场景,如部署在边缘设备的模型。

测试时计算已成为提升大语言模型推理能力的关键范式,但现有方法如Best-of-N和多数投票对所有输入采用统一推理深度,导致简单问题浪费算力,复杂问题可能思考不足。我们提出FR-Ponder,一种单图、无需主干训练的框架,通过潜空间引导实现实例自适应推理计算分配。一个不到100万参数的控制器观察隐藏状态,决定是否终止或添加预计算的引导向量以进行微小推理步进。该方法提取与深层推理输出相关的潜空间引导向量,并通过可调缩放因子重应用,使模型根据输入复杂度自适应调整推理深度。为平衡性能与计算成本,采用组相对策略优化(GRPO)作为奖励信号,动态调节推理深度,实现任务准确率并抑制过度推理。通过课程学习和精细奖励设计,FR-Ponder学会与问题难度相关联的合理算力分配。在GSM8K和MATH500上,该方法显著改善了算力-准确率前沿,以更低的浮点运算次数(FLOPs)达到更匹配的准确率,优于早期退出基线,且不修改主干权重。可视化分析显示可解释的引导方向,表明学习到的算力分配与问题难度高度相关。

原文摘要 · Abstract (English)

Test-time compute has emerged as a key paradigm for enhancing LLM reasoning, yet prevailing approaches like Best-of-N and majority voting apply uniform depth across inputs, wasting computation on simple queries while potentially under-thinking complex ones. We present FR-Ponder, a single-graph, backbone-training-free framework that allocates instance-adaptive reasoning compute via latent steering. A less than 1M-param controller observes hidden states and decides to halt or apply a small ponder step by adding a pre-computed steering vector to frozen representations. Our method extracts the latent steering vector associated with deeper reasoning outputs and direct IO from LLM and re-applies it through a tunable scaling factor, allowing the model to adapt its reasoning depth to the complexity of each input. To balance performance and computational cost, we employ Group Relative Policy Optimization (GRPO) as a reward signal to adaptively regulate reasoning depth, achieving task accuracy while mitigating overreasoning. Through curriculum learning and careful reward engineering, FR-Ponder learns calibrated compute allocation correlated with problem difficulty. On GSM8K and MATH500, FR-Ponder improves the compute-accuracy frontier, delivering lower FLOPs with better matched accuracy and comparing favorably to early-exit baselines, without modifying backbone weights. Analyses visualize interpretable steering directions and show learned compute allocation correlates with problem difficulty.

推理优化潜空间自适应计算大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。