arXiv:2512.21024cs.GTcs.AI2025-12

用代码表示策略,让智能体像人一样理解彼此行为。

Policy-Conditioned Policies for Multi-Agent Task Solving

  • 将策略写成可读代码,通过大模型动态调整应对对手。
  • 在协调博弈和合作觅食任务中达成高效协作。
  • 适合研究多智能体协作与可解释强化学习的人参考。

多智能体任务的核心挑战在于策略的动态适应。然而,在主流深度强化学习范式中,直接基于对手策略进行条件化是不可行的,原因在于存在根本性的“表征瓶颈”:神经网络策略是高维、不可理解的参数向量,其他智能体无法解析。本文提出一种范式转变,通过将策略表示为人类可读的源代码,并利用大型语言模型(LLM)作为近似解释器来弥合这一鸿沟。该程序化表示使我们能够实现博弈论中的‘程序均衡’概念。我们将学习问题重构为在程序化策略空间中直接使用LLM进行优化。LLM充当逐点最优响应算子,迭代合成并精炼自我智能体的策略代码以响应对手策略。我们将其形式化为‘程序化迭代最优响应(PIBR)’算法,通过游戏收益和运行时单元测试生成结构化反馈,以文本梯度优化策略代码。实验表明,该方法有效解决了多个标准协调矩阵博弈及一个合作型分层觅食环境。

原文摘要 · Abstract (English)

In multi-agent tasks, the central challenge lies in the dynamic adaptation of strategies. However, directly conditioning on opponents' strategies is intractable in the prevalent deep reinforcement learning paradigm due to a fundamental ``representational bottleneck'': neural policies are opaque, high-dimensional parameter vectors that are incomprehensible to other agents. In this work, we propose a paradigm shift that bridges this gap by representing policies as human-interpretable source code and utilizing Large Language Models (LLMs) as approximate interpreters. This programmatic representation allows us to operationalize the game-theoretic concept of \textit{Program Equilibrium}. We reformulate the learning problem by utilizing LLMs to perform optimization directly in the space of programmatic policies. The LLM functions as a point-wise best-response operator that iteratively synthesizes and refines the ego agent's policy code to respond to the opponent's strategy. We formalize this process as \textit{Programmatic Iterated Best Response (PIBR)}, an algorithm where the policy code is optimized by textual gradients, using structured feedback derived from game utility and runtime unit tests. We demonstrate that this approach effectively solves several standard coordination matrix games and a cooperative Level-Based Foraging environment.

多智能体策略代码大模型协作

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。