arXiv:2505.13308cs.LGcs.AI2025-05被引 37

用测试时隐空间优化提升大模型推理能力,不改参数也能变聪明。

Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space

  • 在模型隐空间用策略梯度迭代优化推理表示
  • 在GSM8K等数据集上超越思维链和微调方法
  • 只需几次迭代就收敛,适合快速推理场景

推理能力是通向通用人工智能(AGI)的核心挑战。尽管模型性能随训练规模提升,但训练算法存在灾难性遗忘、新数据稀缺等问题。本文提出一种测试时扩展方案:在不更新参数的前提下,通过增加计算量提升推理表现。不同于以往基于标记空间的方法,我们首次在隐空间中实现测试时实例级自适应(TTIA)。提出LatentSeek框架,利用策略梯度对隐状态进行迭代优化,由自生成奖励信号引导。在GSM8K、MATH-500和AIME2024等多个推理基准上评估,涵盖多种大语言模型架构。结果表明,该方法持续优于链式思维提示和微调基线。分析显示,对平均复杂度问题,通常仅需数次迭代即可收敛,且额外迭代仍能带来收益,验证了隐空间测试时扩展的潜力。这为增强大模型推理能力提供了轻量、可扩展、高效的新路径。

原文摘要 · Abstract (English)

Reasoning ability, a core component of human intelligence, continues to pose a significant challenge for Large Language Models (LLMs) in the pursuit of AGI. Although model performance has improved under the training scaling law, significant challenges remain, particularly with respect to training algorithms, such as catastrophic forgetting, and the limited availability of novel training data. As an alternative, test-time scaling enhances reasoning performance by increasing test-time computation without parameter updating. Unlike prior methods in this paradigm focused on token space, we propose leveraging latent space for more effective reasoning and better adherence to the test-time scaling law. We introduce LatentSeek, a novel framework that enhances LLM reasoning through Test-Time Instance-level Adaptation (TTIA) within the model's latent space. Specifically, LatentSeek leverages policy gradient to iteratively update latent representations, guided by self-generated reward signals. LatentSeek is evaluated on a range of reasoning benchmarks, including GSM8K, MATH-500, and AIME2024, across multiple LLM architectures. Results show that LatentSeek consistently outperforms strong baselines, such as Chain-of-Thought prompting and fine-tuning-based methods. Furthermore, our analysis demonstrates that LatentSeek is highly efficient, typically converging within a few iterations for problems of average complexity, while also benefiting from additional iterations, thereby highlighting the potential of test-time scaling in the latent space. These findings position LatentSeek as a lightweight, scalable, and effective solution for enhancing the reasoning capabilities of LLMs.

大模型推理测试时扩展隐空间优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。