测试大模型能否真理解他人想法，发现它常靠记忆而非推理。
Doing Things with Words: Rethinking Theory of Mind Simulation in Large Language Models
- 用模拟人类互动环境测试大模型的共情推理能力。
- 模型在需要推断他人信念时失败率超60%，常靠语言模式猜测。
- 适合关注大模型社会认知局限的研究者与开发者参考。
语言是人类协作的基础，不仅传递信息，还通过共享情境理解协调行动。本研究探讨生成式代理模型Concordia是否能在模拟真实世界环境中有效建模心智理论（ToM）。具体评估GPT-4能否基于社会情境做出真实推断，而非依赖语言记忆完成任务。结果揭示关键局限：GPT-4在需要信念归因时频繁失败，表明此前观察到的类心智理论行为可能源于浅层统计关联，而非真正推理。此外，模型难以生成行动的连贯因果效应，暴露出对复杂社会互动处理困难。这些发现质疑当前关于大模型涌现心智理论能力的说法，强调需建立更严格的、以行为为基础的评估框架。
原文摘要 · Abstract (English)
Language is fundamental to human cooperation, facilitating not only the exchange of information but also the coordination of actions through shared interpretations of situational contexts. This study explores whether the Generative Agent-Based Model (GABM) Concordia can effectively model Theory of Mind (ToM) within simulated real-world environments. Specifically, we assess whether this framework successfully simulates ToM abilities and whether GPT-4 can perform tasks by making genuine inferences from social context, rather than relying on linguistic memorization. Our findings reveal a critical limitation: GPT-4 frequently fails to select actions based on belief attribution, suggesting that apparent ToM-like abilities observed in previous studies may stem from shallow statistical associations rather than true reasoning. Additionally, the model struggles to generate coherent causal effects from agent actions, exposing difficulties in processing complex social interactions. These results challenge current statements about emergent ToM-like capabilities in LLMs and highlight the need for more rigorous, action-based evaluation frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。