构建人类协作中的动作级心理模型数据集，助力智能体理解合作意图。
Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaboration
- 基于经典双人导航任务，标注2987次协作动作的心理模型
- 六款大模型在预测人类行为与心理状态上表现各异
- 适合研究人机协作、认知建模与社会智能的学者使用
近年来，大型语言模型（LLM）代理已具备多步推理、规划和工具使用等复杂认知能力，逐渐成为人类的合作者。然而，有效协作需要双方持续维护并对齐自身推理、对方意图和共同目标的心理模型。当前的代理主要优化任务完成度，缺乏真实的人类协作数据，尤其缺少动作级别的心理模型标注，难以引导代理发展过程级的协作能力。为此，我们提出了ALMANAC——一个基于社会科学研究中经典的双人导航任务（Map Task）构建的动作级心理模型标注数据集。该数据集包含2,987个协作动作，每个动作均配有理论驱动的心理模型标注，涵盖参与者自我推理、对伙伴意图的理解以及对团队目标的感知。我们在六种大语言模型上进行了基准测试，评估其预测人类下一轮行为及心理模型的能力。结果表明，ALMANAC能够有效评估模型模拟人类协作行为及其内在心理机制的能力。
原文摘要 · Abstract (English)
Recent advances in LLM agents have enabled complex cognitive capabilities, such as multi-step reasoning, planning, and tool use, that increasingly position these agents as human collaborators. Effective collaboration, however, requires collaborators to continuously maintain and align mental models of their own reasoning,partners' intentions, and shared goals during the collaborative process. Today's agents rarely develop such capabilities since they are primarily optimized for task completion, and the community lacks authentic human collaboration data with action-level mental model annotations that could guide agents toward process-level collaborative competence. To bridge this gap, we present ALMANAC, a dataset of Action-Level Mental model ANnotations for Agent Collaboration built from the Map Task, a classic dyadic routing task from social science. ALMANAC contains 2,987 collaboration actions, each paired with theory-informed mental model annotations that record the participants' self-reasoning, perceived partner intent, and perceived team goal. We benchmark six LLMs on predicting humans' next-turn behavior and mental models. Our results demonstrate ALMANAC's utility in evaluating models' ability to simulate human collaborative behaviors and infer their underlying mental models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。