简单微调+低秩适配让视觉语言动作模型持续学习效果超预期。
Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning
- 用低秩适配的简单序列微调，实现高效持续学习。
- 在多个终身强化学习基准上表现优于复杂方法，遗忘极小。
- 适合研究大模型持续学习与自适应智能体的学者参考。
视觉语言动作（VLA）模型的持续强化学习是实现可自我提升的具身智能体的重要方向，能适应开放、动态变化的环境。然而传统持续学习观点认为，直接序列微调（Seq. FT）会导致灾难性遗忘，需依赖复杂策略。本文系统研究了大规模预训练VLA在多样化终身强化学习基准上的持续学习表现。结果发现，与既有认知相反，采用低秩适配（LoRA）的简单序贯微调表现优异：具备高可塑性、几乎无遗忘、保留强零样本泛化能力，常超越更复杂的持续学习方法。详细分析表明，其鲁棒性源于大模型预训练、参数高效适配与在线策略强化学习之间的协同作用，重塑了稳定性与可塑性的权衡关系，使持续适应既稳定又可扩展。本工作将序列微调确立为大模型持续强化学习的有效方法，并为大模型时代的终身学习提供了新洞见。代码已开源。
原文摘要 · Abstract (English)
Continual Reinforcement Learning (CRL) for Vision-Language-Action (VLA) models is a promising direction toward self-improving embodied agents that can adapt in openended, evolving environments. However, conventional wisdom from continual learning suggests that naive Sequential Fine-Tuning (Seq. FT) leads to catastrophic forgetting, necessitating complex CRL strategies. In this work, we take a step back and conduct a systematic study of CRL for large pretrained VLAs across diverse lifelong RL benchmarks. We find that, contrary to established belief, simple Seq. FT with low-rank adaptation (LoRA) is remarkably strong: it achieves high plasticity, exhibits little to no forgetting, and retains strong zero-shot generalization, frequently outperforming more sophisticated CRL methods. Through detailed analysis, we show that this robustness arises from a synergy between the large pretrained model, parameter-efficient adaptation, and on-policy RL. Together, these components reshape the stability-plasticity trade-off, making continual adaptation both stable and scalable. Our results position Sequential Fine-Tuning as a powerful method for continual RL with VLAs and provide new insights into lifelong learning in the large model era. Code is available at https://github.com/UT-Austin-RobIn/continual-vla-rl
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。