用强化学习生成复杂心理推理数据，让大模型更懂人心
OSCToM: RL-Guided Adversarial Generation for High-Order Theory of Mind
- 用强化学习和特定语言构造嵌套信念冲突场景
- 在FANToM上达76%准确率，远超旧方法的0.2%
- 适合研究高级认知推理与社会智能的学者
大型语言模型在多数语言任务中表现良好，但在复杂社交情境下的心智理论（ToM）推理仍不均衡。现有基准如ExploreToM未能充分测试递归信念与信息不对称等难点。本文提出OSCToM（观察者-自我冲突心智理论），用于建模基于大模型的ToM任务中的嵌套信念冲突。核心场景是观察者对他人信念的看法与其自身信念状态相冲突，这超越了简单视角转换，需要多层递归推理。OSCToM结合强化学习、扩展领域特定语言和组合代理模型，生成此类冲突案例。实验显示，OSCToM-8B在所有测试系统中表现最优，在FANToM上达到76%准确率，显著优于ExploreToM的0.2%；在Hi-ToM与BigToM上也保持竞争力。数据合成效率提升6倍，表明针对性训练数据可使小模型胜任高阶认知推理。项目代码见https://github.com/sharminsrishty/osct。
原文摘要 · Abstract (English)
Large Language Models (LLMs) perform well on many language tasks, but their Theory of Mind (ToM) reasoning is still uneven in complex social settings. Existing benchmarks, including ExploreToM, do not always test the recursive beliefs and information asymmetries that make these settings difficult. This paper presents OSCToM (Observer-Self Conflict Theory of Mind), an approach for modeling nested belief conflicts in LLM-based ToM tasks. The key case is one in which an observer's view of another agent conflicts with the observer's own belief state. Such cases go beyond simple perspective-taking and require recursive, multi-layered reasoning. OSCToM combines reinforcement learning (RL), an extended domain-specific language, and compositional surrogate models to generate observer-self conflicts. In our experiments, OSCToM-8B gives the best overall result among the systems tested. It improves on the reported ExploreToM results on FANToM and remains competitive on Hi-ToM and BigToM. On the information-asymmetric FANToM benchmark, OSCToM reaches 76% accuracy, compared with the 0.2% reported by ExploreToM. The data-synthesis procedure is also 6x more efficient, indicating that targeted training data can help smaller models handle advanced cognitive reasoning. The project code is available at https://github.com/sharminsrishty/osct.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。