让大模型自己推演状态变化，无需标注数据就能提升推理能力
Language Models can Self-Improve at State-Value Estimation for Better Search
- 用自然语言模拟下一步动作与状态转移，自监督训练价值函数
- 80亿参数模型在网页任务中成功率提升39%，接近闭源模型表现
- 适合想低成本提升推理效率的开发者和研究者
在交互式任务（如网页操作）中，获取真实奖励或人类示范成本高昂。我们提出自我引导前瞻（STL），一种无奖励框架，通过显式推理状态转移来改进基于语言模型的价值函数。STL可视为价值迭代算法的思维链类比：不直接回归数值，而是训练价值语言模型以自然语言预测下一步动作、结果状态及理由，从而在无标签数据下优化价值估计。该自监督过程使状态价值预测更准确，进而让轻量搜索算法在扩展更少节点的同时保持高性能。实证显示，基于中等规模（80亿参数）开源语言模型的STL训练模型，在网页代理任务中成功率提升39%，性能媲美专有模型。STL还适用于多跳问答与数学谜题。我们发现，STL使小型开源模型能高效引导搜索，结合显式推理与价值学习，显著降低推理开销。
原文摘要 · Abstract (English)
Collecting ground-truth rewards or human demonstrations for multi-step reasoning tasks is often prohibitively expensive, particularly in interactive domains such as web tasks. We introduce Self-Taught Lookahead (STL), a reward-free framework that improves language model-based value functions by reasoning explicitly about state transitions. STL can be viewed as a chain-of-thought analogue of the value iteration algorithm: instead of regressing directly on numeric values, a value LLM is trained to simulate a step of lookahead in natural language - predicting the next action, resulting state, and rationale for its value, thereby refining value estimates without any labeled data. This self-supervised procedure yields more accurate state-value predictions, which in turn enable lightweight search algorithms to expand fewer states while maintaining strong performance. Empirically, STL-trained value models built on moderately sized (8B parameter) open-weight LLMs boost web agent success rates by 39%, achieving comparable performance with proprietary models. STL also generalizes to multi-hop QA and math puzzles. We find that STL enables small open-source models to guide efficient search, reducing inference costs by integrating explicit reasoning with value learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。