用游戏竞技场动态评估大模型代码生成能力，发现静态测试的局限性。
ProxyWar: Dynamic Assessment of LLM Code Generation in Game Arenas
- 将大模型生成的代码放入多样的对抗性游戏环境中实测
- 发现模型在真实场景中表现远低于传统基准分数
- 适合关注代码实用性与鲁棒性的研究者和开发者
大型语言模型（LLMs）已革新自动化代码生成，但其在真实世界中的有效性评估仍受限于静态基准和简单指标。我们提出ProxyWar，一种新框架，通过将大模型生成的智能体嵌入多样且具有竞争性的游戏环境，系统评估代码生成质量。不同于现有方法，ProxyWar不仅评估功能正确性，还考察程序的运行特性，结合自动测试、迭代代码修复和多智能体对战，全面揭示程序行为。应用于多种先进编码器和游戏，该方法揭示了基准得分与动态环境下实际表现之间的显著差异，暴露出被忽视的局限与改进空间。这些发现凸显了采用更丰富、基于竞赛的评估方式的必要性。未来，ProxyWar为研究大模型驱动的算法发现、自适应问题求解以及实际效率与鲁棒性提供了基础，甚至可能实现模型超越人工编写的智能体。项目开源地址：https://github.com/xinke-wang/ProxyWar。
原文摘要 · Abstract (English)
Large language models (LLMs) have revolutionized automated code generation, yet the evaluation of their real-world effectiveness remains limited by static benchmarks and simplistic metrics. We present ProxyWar, a novel framework that systematically assesses code generation quality by embedding LLM-generated agents within diverse, competitive game environments. Unlike existing approaches, ProxyWar evaluates not only functional correctness but also the operational characteristics of generated programs, combining automated testing, iterative code repair, and multi-agent tournaments to provide a holistic view of program behavior. Applied to a range of state-of-the-art coders and games, our approach uncovers notable discrepancies between benchmark scores and actual performance in dynamic settings, revealing overlooked limitations and opportunities for improvement. These findings highlight the need for richer, competition-based evaluation of code generation. Looking forward, ProxyWar lays a foundation for research into LLM-driven algorithm discovery, adaptive problem solving, and the study of practical efficiency and robustness, including the potential for models to outperform hand-crafted agents. The project is available at https://github.com/xinke-wang/ProxyWar.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。