测试大模型能否像人一样通过推理他人想法来规划说服策略
Do Large Language Models Have a Planning Theory of Mind? Evidence from MindGames: a Multi-Step Persuasion Task
- 设计新任务让模型推断对方信念与意图以改变其行为
- 人类在复杂心理推理任务中比o1-preview高11%（p=0.006）
- 适合研究人机社会智能差异的学者关注
最新研究表明大语言模型具备心智理论（ToM）能力。但多数实验仅让参与者被动预测他人行为，而人类的ToM还体现在主动规划并干预他人心理状态。本文提出新任务MindGames，考察模型在多步说服任务中基于对方信念与欲望进行策略规划的能力。结果发现，人类在该任务中显著优于o1-preview（准确率高11%，p=0.006）。我们推测这是因为人类拥有对他人行为的隐式因果模型（如知道需先询问偏好）。相反，在无需心理推理、仅需规划的任务中，o1-preview表现优于人类。这表明当前大模型在类人社会推理方面仍存在显著差距。
原文摘要 · Abstract (English)
Recent evidence suggests Large Language Models (LLMs) display Theory of Mind (ToM) abilities. Most ToM experiments place participants in a spectatorial role, wherein they predict and interpret other agents' behavior. However, human ToM also contributes to dynamically planning action and strategically intervening on others' mental states. We present MindGames: a novel `planning theory of mind' (PToM) task which requires agents to infer an interlocutor's beliefs and desires to persuade them to alter their behavior. Unlike previous evaluations, we explicitly evaluate use cases of ToM. We find that humans significantly outperform o1-preview (an LLM) at our PToM task (11% higher; $p=0.006$). We hypothesize this is because humans have an implicit causal model of other agents (e.g., they know, as our task requires, to ask about people's preferences). In contrast, o1-preview outperforms humans in a baseline condition which requires a similar amount of planning but minimal mental state inferences (e.g., o1-preview is better than humans at planning when already given someone's preferences). These results suggest a significant gap between human-like social reasoning and LLM abilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。