用搜索优化多轮强化学习的轨迹生成，提升智能体表现与训练稳定性。
TSR: Trajectory-Search Rollouts for Multi-Turn RL of LLM Agents
- 在每轮交互中通过轻量树搜索选择高分动作，生成更优轨迹
- 在Sokoban等任务上提升15%性能，且学习过程更稳定
- 兼容主流优化器，适合想改进多轮智能体训练的研究者
大型语言模型的发展正推动利用强化学习从跨任务的迭代多轮交互中训练智能体。然而，多轮强化学习仍具挑战性，因奖励稀疏或延迟，环境具有随机性。朴素轨迹采样易阻碍利用并导致模式崩溃。我们提出TSR（轨迹搜索播送），一种训练时方法，将测试时缩放思想重用于提升每轮播送质量。TSR通过基于状态的反馈，在每轮中以轻量树搜索选择高分动作，构建高质量轨迹。该方法提升了播送质量并稳定了学习，同时兼容标准策略梯度优化器，实现优化器无关。我们以best-of-N、束搜索和浅层前瞻搜索实现TSR，搭配PPO和GRPO，在Sokoban、FrozenLake和WebShop任务上实现最高15%性能提升，仅需一次性的训练计算增加。通过将搜索从推理阶段移至训练播送阶段，TSR提供了一种模块化且通用的强多轮智能体学习机制，可补充现有框架及拒绝采样类选择方法。
原文摘要 · Abstract (English)
Advances in large language models (LLMs) are driving a shift toward using reinforcement learning (RL) to train agents from iterative, multi-turn interactions across tasks. However, multi-turn RL remains challenging as rewards are often sparse or delayed, and environments can be stochastic. In this regime, naive trajectory sampling can hinder exploitation and induce mode collapse. We propose TSR (Trajectory-Search Rollouts), a training-time approach that repurposes test-time scaling ideas for improved per-turn rollout generation. TSR performs lightweight tree-style search to construct high-quality trajectories by selecting high-scoring actions at each turn using state-based feedback. This improves rollout quality and stabilizes learning while remaining compatible with standard policy gradient optimizers, making TSR optimizer-agnostic. We instantiate TSR with best-of-N, beam, and shallow lookahead search, and pair it with PPO and GRPO, achieving up to 15% performance gains and more stable learning on Sokoban, FrozenLake, and WebShop tasks at a modest, one-time increase in training compute. By moving search from inference time to the rollout stage of training, TSR provides a modular and general mechanism for stronger multi-turn agent learning, complementary to existing frameworks and rejection-sampling-style selection methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。