让大模型在部署中持续学习，用经验回放提升样本效率。
CLaaS: Continual learning as a service for sample efficient online learning
- 通过经验回放缓冲区实现异步训练中的梯度复用。
- 相比上下文学习，参数更新显著减少遗忘并提升迁移效果。
- 适合需要动态适应的实时部署场景，如智能客服或自动驾驶。
已部署的大语言模型智能体需适应动态环境中的分布漂移。理想情况下，智能体应能基于累积的经验进行适应，同时保留已有能力并迁移到未来任务。然而，由于真实环境无法轻易重置，每个场景只能采样一次。为此，我们研究了一种基于经验的在线持续学习设置，其中智能体从连续场景流中学习。我们提出持续学习即服务（CLaaS），一个可抽象为聊天API的系统，使智能体能在部署过程中持续改进。为提高样本效率，CLaaS将轨迹存储在经验回放缓冲区中，用于异步训练时的梯度复用。我们在对抗性任务上评估了CLaaS，结果表明参数更新相比上下文学习能实现更好的前向迁移且遗忘更少，而经验回放是提升样本效率的关键因素。
原文摘要 · Abstract (English)
Deployed large language model agents must adapt to distribution shift in dynamic environments. Ideally, adaptation can be performed from accumulated agent experiences and retain prior capabilities while transferring to future tasks. However, agent actions and environmental transitions can only be sampled once per scenario, as real-world environments cannot be trivially reset. To this end, we investigate an experiential and online continual learning setting in which agents learn from a stream of scenarios. We propose continual learning as-a-service (CLaaS), a system which enables agents to improve during deployment, abstracted behind a chat API. To increase sample efficiency, CLaaS stores rollouts in an experience replay buffer for gradient reuse during asynchronous training. We evaluate CLaaS on an adversarial task, demonstrating that parametric updates lead to superior forward transfer and less forgetting than in-context learning, with replay being a critical choice for sample efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。