让大模型当观察者,看懂对手的出招策略。
Observer, Not Player: Simulating Theory of Mind in LLMs through Game Observation
- 大模型以观察者身份分析博弈策略,不直接参与
- 提出联合损失函数,量化策略识别准确率与推理合理性
- 支持实时交互调试,适合研究模型推理能力的学者
我们提出一个交互式框架,用于评估大语言模型(LLMs)在简单但具战略性的环境中是否表现出真正的“理解”。以剪刀石头布(RPS)为例,尽管规则简单,却需要序列推理、适应性和策略识别。系统将大模型定位为观察者,任务是识别正在使用的策略并阐述判断依据。重点不在于测试对游戏规则的掌握,而是探测模型能否对序列行为进行类心智推理。为支持系统性评估,我们构建了一个基准,包含静态策略和由良好提示定义的轻量级动态策略。通过交叉熵、Brier得分和期望收益(EV)偏差三种互补信号,量化观察者预测与实际策略组合引发的真实分布之间的对齐程度。这些指标整合为统一的联合损失(Union Loss),平衡校准性、敏感性和收益对齐。结合策略识别率(SIR)指标,该框架不仅衡量预测准确性,还检验模型是否能稳定识别隐藏策略。演示强调交互性、透明性和可复现性:用户可实时调整模型分布,可视化损失变化,并直接检查推理片段以定位失败原因。整体系统为序列博弈中的类心智推断提供了实用且可解释的代理,揭示当前大模型推理的优势与局限。
原文摘要 · Abstract (English)
We present an interactive framework for evaluating whether large language models (LLMs) exhibit genuine "understanding" in a simple yet strategic environment. As a running example, we focus on Rock-Paper-Scissors (RPS), which, despite its apparent simplicity, requires sequential reasoning, adaptation, and strategy recognition. Our system positions the LLM as an Observer whose task is to identify which strategies are being played and to articulate the reasoning behind this judgment. The purpose is not to test knowledge of Rock-Paper-Scissors itself, but to probe whether the model can exhibit mind-like reasoning about sequential behavior. To support systematic evaluation, we provide a benchmark consisting of both static strategies and lightweight dynamic strategies specified by well-prompted rules. We quantify alignment between the Observer's predictions and the ground-truth distributions induced by actual strategy pairs using three complementary signals: Cross-Entropy, Brier score, and Expected Value (EV) discrepancy. These metrics are further integrated into a unified score, the Union Loss, which balances calibration, sensitivity, and payoff alignment. Together with a Strategy Identification Rate (SIR) metric, our framework captures not only predictive accuracy but also whether the model can stably identify the latent strategies in play. The demo emphasizes interactivity, transparency, and reproducibility. Users can adjust LLM distributions in real time, visualize losses as they evolve, and directly inspect reasoning snippets to identify where and why failures occur. In doing so, our system provides a practical and interpretable proxy for mind-like inference in sequential games, offering insights into both the strengths and limitations of current LLM reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。