让视觉语言助手通过强化学习高效训练，小显卡也能跑大模型。
RLLaVA: An RL-central Framework for Language and Vision Assistants
- 将强化学习逻辑与模型架构分离，支持快速接入新算法和模型。
- 10亿到70亿参数模型可在普通显卡上高效训练，40亿模型单卡可全参数更新。
- 适用于多模态任务和智能体应用，适合想快速实验RL方法的研究者。
我们提出一种以强化学习为核心的视觉语言助手框架RLLaVA，其基于马尔可夫决策过程（MDP）建模。RLLaVA将强化学习算法逻辑与模型结构、分布式执行解耦，使研究者仅需少量代码即可实现新算法，并灵活集成多种强化学习方法与视觉语言模型（VLMs），且不依赖特定训练或推理引擎。该框架使得在普通GPU上实现10亿至70亿参数模型的资源高效训练成为可能；尤其值得注意的是，40亿规模的模型可在单张24GB GPU上完成端到端的全参数更新训练。在多模态与智能体任务上的实验表明，RLLaVA具备良好的任务可扩展性，所训练模型性能持续优于基础模型，达到与其他专门设计的强化学习框架相当的水平。代码已开源：https://github.com/TinyLoopX/RLLaVA。
原文摘要 · Abstract (English)
We present an RL-central framework for Language and Vision Assistants (RLLaVA) with its formulation of Markov decision process (MDP). RLLaVA decouples RL algorithmic logic from model architecture and distributed execution, supporting researchers in implementing new RL algorithms with minimal code, and to plug in a broad family of RL methods and vision-language models (VLMs) while remaining agnostic to specific training and inference engines. RLLaVA makes resource-efficient training of 1B--7B models feasible on common GPUs; notably, 4B-scale models can be trained end-to-end with full-parameter updates on a single 24GB GPU. Experiments on multi-modal and agentic tasks demonstrate that RLLaVA has task extensibility, and the models trained with it consistently improve performance over base models, competitive with other specially engineered RL frameworks. The code is available at https://github.com/TinyLoopX/RLLaVA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。