用预训练视觉语言模型做多智能体策略价值评估，提升采样效率。
MA-VLCM: A Vision Language Critic Model for Value Estimation of Policies in Multi-Agent Team Settings
- 用预训练视觉语言模型替代传统学习的中心化价值网络。
- 在多智能体任务中实现零样本回报估计，跨环境泛化能力强。
- 适合资源受限机器人部署，支持不同硬件配置的多机协同。
多智能体强化学习常依赖中心化价值函数估计器，但从头学习效率低且泛化能力差。大规模视觉-语言-动作模型（VLAs）虽具备强大的多模态推理和零样本泛化能力，却难以直接用于机器人执行，尤其在异构多机器人系统中计算开销大。为此，我们提出多智能体视觉-语言-批评者模型（MA-VLCM），将MARL中的学习型中心化批评者替换为微调后的预训练视觉语言模型，以自然语言任务描述、视觉轨迹观测和结构化多智能体状态为输入，实现条件化的价值估计。该方法避免了策略优化过程中的批评者学习，显著提升采样效率，并生成适于资源受限机器人部署的紧凑策略。实验表明，不同视觉语言模型主干在分布内与分布外场景下均表现出良好的零样本回报估计性能。
原文摘要 · Abstract (English)
Multi-agent reinforcement learning (MARL) commonly relies on a centralized critic to estimate the value function. However, learning such a critic from scratch is highly sample-inefficient and often lacks generalization across environments. At the same time, large vision-language-action models (VLAs) trained on internet-scale data exhibit strong multimodal reasoning and zero-shot generalization capabilities, yet directly deploying them for robotic execution remains computationally prohibitive, particularly in heterogeneous multi-robot systems with diverse embodiments and resource constraints. To address these challenges, we propose Multi-Agent Vision-Language-Critic Models (MA-VLCM), a framework that replaces the learned centralized critic in MARL with a pretrained vision-language model fine-tuned to evaluate multi-agent behavior. MA-VLCM acts as a centralized critic conditioned on natural language task descriptions, visual trajectory observations, and structured multi-agent state information. By eliminating critic learning during policy optimization, our approach significantly improves sample efficiency while producing compact execution policies suitable for deployment on resource-constrained robots. Results show good zero-shot return estimation on models with differing VLM backbones on in-distribution and out-of-distribution scenarios in multi-agent team settings
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。