用小数据微调LLM，让AI在外交游戏中学会博弈策略。
DipLLM: Fine-Tuning LLM for Strategic Decision-making in Diplomacy
- 将多单位行动分解为逐单位序列决策，简化复杂博弈
- 仅用1.5%数据就超越顶尖模型Cicero的表现
- 适合研究多智能体博弈与小样本策略学习的学者
外交是一款复杂的多人博弈游戏，兼具合作与竞争，对AI系统构成重大挑战。传统方法依赖均衡搜索生成大量训练数据，计算成本高。大语言模型（LLMs）可借助预训练知识，以较小规模微调实现优异表现，但应用于外交游戏仍面临行动组合爆炸及玩家间复杂互动难题。为此，我们提出DipLLM，一种基于微调LLM的代理，学习外交游戏中的均衡策略。DipLLM采用自回归因子化框架，将复杂的多单位行动分配任务转化为一系列单位级决策。在该框架内定义均衡策略作为学习目标，仅使用状态领先模型Cicero所需数据的1.5%，即实现超越其性能。结果表明，微调后的LLM在复杂多玩家博弈中具有解决战略决策问题的巨大潜力。
原文摘要 · Abstract (English)
Diplomacy is a complex multiplayer game that requires both cooperation and competition, posing significant challenges for AI systems. Traditional methods rely on equilibrium search to generate extensive game data for training, which demands substantial computational resources. Large Language Models (LLMs) offer a promising alternative, leveraging pre-trained knowledge to achieve strong performance with relatively small-scale fine-tuning. However, applying LLMs to Diplomacy remains challenging due to the exponential growth of possible action combinations and the intricate strategic interactions among players. To address this challenge, we propose DipLLM, a fine-tuned LLM-based agent that learns equilibrium policies for Diplomacy. DipLLM employs an autoregressive factorization framework to simplify the complex task of multi-unit action assignment into a sequence of unit-level decisions. By defining an equilibrium policy within this framework as the learning objective, we fine-tune the model using only 1.5% of the data required by the state-of-the-art Cicero model, surpassing its performance. Our results demonstrate the potential of fine-tuned LLMs for tackling complex strategic decision-making in multiplayer games.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。