让大模型在推理时动态优化隐向量，提升复杂任务准确率
Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization
- 测试时通过梯度优化隐式思维向量，不更新模型参数
- 在AIME难题上将准确率从接近0提升至显著水平
- 无需外部标注，仅用模型自身置信度生成奖励信号
大型语言模型（LLMs）近期从显式的思维链（CoT）推理转向更高效的隐式推理，即用向量表示中间思考过程而非文本。然而，在挑战性、分布外的任务上，隐式推理往往表现脆弱，而这些场景正是需要强推理能力的关键。为此，本文提出测试时隐式思维策略优化（LTPO），一种完全在测试阶段运行、无需更新模型参数的增强框架。LTPO将中间隐式‘思维’向量视为可动态优化的变量，采用基于模型自身输出分布的内在置信度奖励信号，通过在线策略梯度方法进行优化，避免了外部监督或昂贵的文本生成。在五个推理基准上的大量实验表明，LTPO不仅在标准任务上达到或超越强基线，且在其他方法失效时仍表现出显著鲁棒性。尤为突出的是，在极具挑战性的AIME基准上，现有隐式推理基线准确率接近零，而LTPO实现显著提升，展现出处理复杂推理的独特能力。
原文摘要 · Abstract (English)
Recent advancements in Large Language Models (LLMs) have shifted from explicit Chain-of-Thought (CoT) reasoning to more efficient latent reasoning, where intermediate thoughts are represented as vectors rather than text. However, latent reasoning can be brittle on challenging, out-of-distribution tasks where robust reasoning is most critical. To overcome these limitations, we introduce Latent Thought Policy Optimization (LTPO), a parameter-free framework that enhances LLM reasoning entirely at test time, without requiring model parameter updates. LTPO treats intermediate latent "thought" vectors as dynamic parameters that are actively optimized for each problem instance. It employs an online policy gradient method guided by an intrinsic, confidence-based reward signal computed directly from the frozen LLM's own output distributions, eliminating the need for external supervision or expensive text generation during optimization. Extensive experiments on five reasoning benchmarks show that LTPO not only matches or surpasses strong baselines on standard tasks but also demonstrates remarkable robustness where others fail. Most notably, on highly challenging AIME benchmarks where existing latent reasoning baselines collapse to near-zero accuracy, LTPO delivers substantial improvements, showcasing a unique capability for complex reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。