给大模型加记忆模块，提升博弈类任务的连续决策能力
Towards Improving Sequential Decision-Making in LLM Agents via Experience Memory
- 引入经验记忆机制，解决序列决策中的信用分配难题
- 反思与规则提取使井字棋表现显著提升，无需改模型权重
- 在零和博弈中验证有效，适合研究智能体决策的学者
大型语言模型在单次推理任务上进步显著，但在序列决策任务上的表现仍不清晰。我们以完全可观测的双人零和博弈为研究场景，提供真实评估：结果由规则决定，单步最优性可计算或近似，无需依赖裁判模型。跨不同模型层级，LLM在井字棋、连珠等简单游戏中表现不佳，且无法战胜基于蒙特卡洛树搜索（MCTS）的对手。改变游戏树的表面形式但保留其结构，性能基本不变，表明差距并非主要源于记忆策略的缺失。为此，我们提出一种增强经验记忆的智能体框架，专为序列决策设计，解决信用分配等常见挑战。实验表明，赛后反思与规则提取可在不修改模型权重的前提下，显著提升井字棋的表现。
原文摘要 · Abstract (English)
Large language models have improved substantially on single-shot reasoning tasks, but their performance in sequential decision-making is less well understood. We study this on fully-observable two-player zero-sum games, which provide ground-truth evaluation: outcomes are determined by the rules, and optimality of individual moves can be computed or approximated, without relying on a judge model. Across model tiers, LLMs play suboptimally in simple games such as tic-tac-toe or Connect Four, and lose to MCTS opponents. Obfuscations that preserve the game tree but rewrite its surface form leave performance largely unchanged, indicating the gap is not fully explained by recall of memorized strategies. Motivated by this performance gap, we introduce an agentic framework enhanced with an experience memory designed for the sequential setting and addressing common challenges of sequential decision-making such as credit assignment. We show that post-game reflection and rule extraction yield measurable improvements on tic-tac-toe without modifying the model weights.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。