AVACraft让视觉语言模型直接玩星际争霸，零样本胜率超75%
AVA: Attentive VLM Agent for Mastering StarCraft II
- 构建多模态环境，支持视觉+语言+结构化状态的联合训练
- 零样本下VLM模型胜率达75%-90%，远超训练后MARL的19.3%
- 适合研究AI决策可解释性与零样本部署的团队
我们提出AVACraft，一个支持多智能体强化学习（MARL）与视觉语言模型（VLM）的星际争霸II基准。不同于依赖抽象状态表示的SMAC类环境，AVACraft提供RGB画面、自然语言观测和结构化状态信息，可在21个涵盖微操、协同与战略规划的场景中系统比较基于训练与零样本方法。我们建立全面基线：六种MARL算法（IQL, QMIX, QTRAN, VDN, MAPPO, IPPO）采用Swin-Transformer骨干网络训练500万步；多种VLM包括专有模型（GPT-4o）与开源模型（Qwen3-VL）。结果表明互补优势：训练后的MARL在500万步后达到19.3%胜率，而VLM实现75%-90%零样本胜率且决策更符合人类意图，揭示了训练效率、性能上限、可解释性与部署成本之间的权衡。代码已开源。
原文摘要 · Abstract (English)
We introduce AVACraft, a multimodal StarCraft II benchmark supporting both Multi-Agent Reinforcement Learning (MARL) and Vision-Language Model (VLM) paradigms. Unlike SMAC-family environments that rely on abstract state representations and exclude VLMs, AVACraft provides RGB visuals, natural language observations, and structured state information, enabling systematic comparison between training-based and zero-shot methods across 21 scenarios spanning micromanagement, coordination, and strategic planning. We establish comprehensive baselines: six MARL algorithms (IQL, QMIX, QTRAN, VDN, MAPPO, IPPO) with Swin-Transformer backbones trained for 5M steps, and multiple VLMs including proprietary (GPT-4o) and open-source (Qwen3-VL) models. Results reveal complementary strengths-MARL peaks at 19.3% win rate after 5M steps, while VLMs achieve 75-90% zero-shot with human-aligned decisions-exposing trade-offs between training efficiency, performance ceilings, interpretability, and deployment cost. Code: https://github.com/camel-ai/VLM-Play-StarCraft2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。