LLM在动态博弈中表现受状态表述方式影响,简洁、含悔恨信息的表述更接近最优策略。
The Effect of State Representation on LLM Agent Behavior in Dynamic Routing Games
- 从动作、收益和提示风格三方面系统构建自然语言状态表示框架
- 含悔恨信息且摘要化的历史表述使代理行为更贴近理论均衡,波动更小
- 适合研究大模型决策机制或博弈设计的研究者参考
大型语言模型(LLMs)在动态环境中展现出作为决策者的潜力,但其无状态特性要求将历史信息转化为自然语言表示。本文提出一个统一框架,系统构建用于重复多智能体博弈中提示LLM代理的自然语言“状态”表示。以往研究对游戏历史的编码方式随意,不仅掩盖了状态表示对代理行为的影响,也限制了不同研究间的可比性。本框架从三个维度刻画状态表示方法:动作信息量(是否捕捉已执行动作)、收益信息量(是否描述获得收益)以及提示风格(自然语言压缩程度,即历史文本的摘要程度)。我们将其应用于一个动态自私路由博弈,该博弈在理论上和人类实验中均有简单均衡解 \\cite{rapoport_choice_2009}。尽管博弈本身较简单,我们仍发现LLM代理行为显著依赖于自然语言状态表示。具体而言,提供(1)摘要化而非完整的历史表述;(2)悔恨信息而非原始收益;(3)有限他人动作信息的表示,能促使代理行为更接近博弈论均衡预测,且游戏行为更稳定。相反,其他表示方式可能导致与均衡的大幅偏离,或随时间变化更大的动态波动。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown promise as decision-makers in dynamic settings, but their stateless nature necessitates creating a natural language representation of history. We present a unifying framework for systematically constructing natural language "state" representations for prompting LLM agents in repeated multi-agent games. Previous work on games with LLM agents has taken an ad hoc approach to encoding game history, which not only obscures the impact of state representation on agents' behavior, but also limits comparability between studies. Our framework addresses these gaps by characterizing methods of state representation along three axes: action informativeness (i.e., the extent to which the state representation captures actions played); reward informativeness (i.e., the extent to which the state representation describes rewards obtained); and prompting style (or natural language compression, i.e., the extent to which the full text history is summarized). We apply this framework to a dynamic selfish routing game, chosen because it admits a simple equilibrium both in theory and in human subject experiments \cite{rapoport_choice_2009}. Despite the game's relative simplicity, we find that there are key dependencies of LLM agent behavior on the natural language state representation. In particular, we observe that representations which provide agents with (1) summarized, rather than complete, natural language representations of past history; (2) information about regrets, rather than raw payoffs; and (3) limited information about others' actions lead to behavior that more closely matches game theoretic equilibrium predictions, and with more stable game play by the agents. By contrast, other representations can exhibit either large deviations from equilibrium, higher variation in dynamic game play over time, or both.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。