无需梯度更新，让大模型在运行时实时优化决策。
Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates
- 用动态记忆检索历史轨迹，实时估算动作优势并直接调节模型输出。
- 在WebArena和Jericho上超越现有免训练方法，性能接近微调效果。
- 成本低于微调30倍，适合需持续学习的低成本部署场景。
大型语言模型（LLM）代理在通用任务中表现优异，但部署后权重冻结导致难以持续适应。传统强化学习虽可解决此问题，但计算开销大且易引发灾难性遗忘。本文提出即时强化学习（JitRL），一种无需训练的框架，可在测试阶段实现无梯度更新的策略优化。JitRL维护一个非参数化的动态经验记忆，实时检索相关轨迹以估计动作优势，并直接调整LLM输出的logits。理论证明该加性更新规则是KL约束策略优化目标的精确闭式解。在WebArena和Jericho上的实验表明，JitRL在免训练方法中达到新基准，性能优于计算昂贵的微调方法（如WebRL），同时降低超30倍的金钱成本，为持续学习代理提供了可扩展路径。代码已开源：https://github.com/liushiliushi/JitRL。
原文摘要 · Abstract (English)
While Large Language Model (LLM) agents excel at general tasks, they inherently struggle with continual adaptation due to the frozen weights after deployment. Conventional reinforcement learning (RL) offers a solution but incurs prohibitive computational costs and the risk of catastrophic forgetting. We introduce Just-In-Time Reinforcement Learning (JitRL), a training-free framework that enables test-time policy optimization without any gradient updates. JitRL maintains a dynamic, non-parametric memory of experiences and retrieves relevant trajectories to estimate action advantages on-the-fly. These estimates are then used to directly modulate the LLM's output logits. We theoretically prove that this additive update rule is the exact closed-form solution to the KL-constrained policy optimization objective. Extensive experiments on WebArena and Jericho demonstrate that JitRL establishes a new state-of-the-art among training-free methods. Crucially, JitRL outperforms the performance of computationally expensive fine-tuning methods (e.g., WebRL) while reducing monetary costs by over 30 times, offering a scalable path for continual learning agents. The code is available at https://github.com/liushiliushi/JitRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。