用强化学习让视觉语言模型学会交互决策，自改进并适应严格输出要求。
VLM Q-Learning: Aligning Vision-Language Models for Interactive Decision-Making
- 基于离线到在线的强化学习，让视觉语言模型从自身错误中学习。
- 在三个多模态任务中提升模型对严格输出格式的遵循能力。
- 适合需要视觉推理与交互决策的自动化应用开发者。
近期研究尝试将大语言模型（LLMs）的通用知识与推理能力融入交互环境中的智能体，以完成用户指定目标。视觉语言模型（VLMs）扩展了LLMs对多模态数据的支持，使智能体具备视觉推理能力，适用于计算机自动化等新场景。然而，现有开放权重的VLM在代理任务上仍落后于其语言模型对应物，例如难以遵循环境严格的输出语法，且更偏向开放式问答。克服这些限制需在特定任务专家示范上进行监督微调（SFT）。本文从离线到在线强化学习（RL）视角出发，利用RL在不依赖高质量数据的情况下，让智能体从自身或更大模型的失败决策中学习。我们提出一种非策略性强化学习方案，在保持广泛使用的SFT流程稳定性与简便性的基础上，实现智能体自迭代与低质量数据学习。我们在两个开源权重的VLM上，验证了该方法在三个多模态代理领域中的有效性。
原文摘要 · Abstract (English)
Recent research looks to harness the general knowledge and reasoning of large language models (LLMs) into agents that accomplish user-specified goals in interactive environments. Vision-language models (VLMs) extend LLMs to multi-modal data and provide agents with the visual reasoning necessary for new applications in areas such as computer automation. However, agent tasks emphasize skills where accessible open-weight VLMs lag behind their LLM equivalents. For example, VLMs are less capable of following an environment's strict output syntax requirements and are more focused on open-ended question answering. Overcoming these limitations requires supervised fine-tuning (SFT) on task-specific expert demonstrations. Our work approaches these challenges from an offline-to-online reinforcement learning (RL) perspective. RL lets us fine-tune VLMs to agent tasks while learning from the unsuccessful decisions of our own model or more capable (larger) models. We explore an off-policy RL solution that retains the stability and simplicity of the widely used SFT workflow while allowing our agent to self-improve and learn from low-quality datasets. We demonstrate this technique with two open-weight VLMs across three multi-modal agent domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。