让大模型在星际争霸中用自然语言协作，测试真实团队配合能力
SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models
- 引入自然语言通信通道，模拟真实多智能体协作场景
- 设计欺骗性通信者，检验模型在干扰下的信任与决策能力
- 基于通义千问系列模型，研究推理结构对协作的影响
随着大语言模型（LLM）应用日益广泛，它们被期望能与其他AI智能体协同工作，而非孤立运行。有效协作需要智能体在不确定性下进行沟通、共享信息并做出决策。本文提出SMAC-Talk，是星灵多智能体挑战（StarCraft Multi-Agent Challenge, SMAC）的自然语言扩展，用于评估基于大语言模型的智能体在合作多智能体环境中的表现。该环境具备去中心化控制、部分可观测性和长时程决策等特征。SMAC-Talk引入自然语言通信通道，用于探测智能体间的协调与信任机制。我们利用该通道构建多种评估场景，包括嵌入式欺骗性通信者，仅通过语言干扰盟友。我们使用4个通义千问3.5系列模型作为基准，评估不同推理结构、记忆机制和模型规模对智能体协作的影响。SMAC-Talk已开源，以支持社区在合作多智能体设置中开发和评估大语言模型智能体。
原文摘要 · Abstract (English)
As LLMs become more widely deployed, they are increasingly expected to work alongside other AI agents rather than operating in isolation. Effective coordination in these settings requires agents to communicate, share information and make decisions under uncertainty. We introduce SMAC-Talk, a natural language extension of the StarCraft Multi-Agent Challenge for evaluating LLM-based agents in cooperative multi-agent environments. The environment has several key features such as decentralized control, partial observability and long-horizon decision making. SMAC-Talk includes a natural language communication channel which is used to probe agent coordination and trust. We use this communication channel to construct different evaluation scenarios, including settings with an embedded deceptive communicator that tries to disrupt and deceive allies through communication alone. We provide three agents for benchmarking using 4 models from the Qwen3.5 family and study how reasoning structure, memory and model scale affect coordination between agents. We release SMAC-Talk as an open benchmark to support the research community in developing and evaluating LLM agents in cooperative multi-agent settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。