用视觉语言模型自动筛选优质经验，提升强化学习效率与成功率。
VLM-Guided Experience Replay
- 用预训练的VLM评估并优先存储高价值子轨迹
- 游戏与机器人任务中成功率达11%-52%提升，采样效率提高19%-45%
- 无需微调，适配多领域，适合追求高效学习的RL研究者
大语言模型（LLMs）和视觉语言模型（VLMs）的进展带来了强大的语义与多模态推理能力，为增强强化学习（RL）中的样本效率、高层规划与可解释性提供了新机遇。尽管已有工作将LLMs和VLMs融入RL的多个组件，但作为存储与重用经验的核心组件——经验回放缓冲区仍缺乏探索。本文提出利用VLM引导回放缓冲区中的经验优先级。核心思路是使用一个冻结的预训练VLM（无需微调）作为自动化评估器，识别并优先处理智能体经验中的有前景子轨迹。在涵盖离散与连续动作空间的游戏与机器人任务中，采用该方法训练的智能体相比以往方法平均成功率提升11%-52%，样本效率提高19%-45%。
原文摘要 · Abstract (English)
Recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) have enabled powerful semantic and multimodal reasoning capabilities, creating new opportunities to enhance sample efficiency, high-level planning, and interpretability in reinforcement learning (RL). While prior work has integrated LLMs and VLMs into various components of RL, the replay buffer, a core component for storing and reusing experiences, remains unexplored. We propose addressing this gap by leveraging VLMs to guide the prioritization of experiences in the replay buffer. Our key idea is to use a frozen, pre-trained VLM (requiring no fine-tuning) as an automated evaluator to identify and prioritize promising sub-trajectories from the agent's experiences. Across scenarios, including game-playing and robotics, spanning both discrete and continuous domains, agents trained with our proposed prioritization method achieve 11-52% higher average success rates and improve sample efficiency by 19-45% compared to previous approaches. https://esharony.me/projects/vlm-rb/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。