无需微调，任何大模型都能玩完整版外交游戏。
Democratizing Diplomacy: A Harness for Evaluating Any Large Language Model on Full-Press Diplomacy
- 用数据驱动优化文本状态表示，让24B模型无须训练即可对战。
- 24B模型可稳定完成对局，小模型表现也达标。
- 提供深度分析工具，适合研究战略推理与说服机制。
我们提出首个评估框架，使任意本地部署的大语言模型（LLM）可在不进行微调或专项训练的情况下，参与完整版外交游戏。此前研究受限于外交游戏状态的高度复杂性与信息密度，需依赖前沿模型或微调，且比赛结果方差大，难以研究。本文通过数据驱动迭代优化文本状态表示，使240亿参数模型在无微调条件下稳定完成对局。我们开发了支持假设检验与统计分析的工具，并开展说服力、激进策略及多模型性能对比的案例研究。实验覆盖多个主流LLM，发现大模型表现更优，但小模型仍具可用性。此外，我们提出关键状态分析（Critical State Analysis），实现对对局关键节点的快速迭代与深度剖析。该框架消除了微调需求，推动了对大模型战略推理能力的普适评估，并揭示这些能力如何自然涌现于通用模型中。代码已开源。
原文摘要 · Abstract (English)
We present the first evaluation harness that enables any out-of-the-box, local, Large Language Models (LLMs) to play full-press Diplomacy without fine-tuning or specialized training. Previous work required frontier LLMs, or fine-tuning, due to the high complexity and information density of Diplomacy's game state. Combined with the high variance of matches, these factors made Diplomacy prohibitive for study. In this work, we used data-driven iteration to optimize a textual game state representation such that a 24B model can reliably complete matches without any fine tuning. We develop tooling to facilitate hypothesis testing and statistical analysis, and we present case studies on persuasion, aggressive playstyles, and performance across a range of models. We conduct a variety of experiments across many popular LLMs, finding the larger models perform the best, but the smaller models still play adequately. We also introduce Critical State Analysis: an experimental protocol for rapidly iterating and analyzing key moments in a game at depth. Our harness democratizes the evaluation of strategic reasoning in LLMs by eliminating the need for fine-tuning, and it provides insights into how these capabilities emerge naturally from widely used LLMs. Our code is available in the supplement and will be open sourced.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。