用认知模型追踪对话中用户心理变化,更好评估聊天机器人表现
Cognitive World Model for Progressive BDI/E Trajectory Evaluation of Conversational Agents

- 构建基于大模型的用户认知状态模拟器,同步跟踪信念、欲望、意图和情绪
- 在15万轮对话数据上训练,对用户反应和心理状态预测更准确
- 适合研究对话策略优化与人机交互心理影响的学者与开发者
随着基于大模型的对话系统向开放、高互动场景演进,仅关注任务完成度无法全面评估其效果。用户内在状态(信念、欲望、意图、情绪,简称BDI/E)是连接行为与交互结果的中间信号,反映对话策略如何在多轮交互中塑造用户心理。然而现有评估方法多聚焦表面回复或最终结果,难以揭示背后认知过程,导致难以诊断系统成败并优化策略。为此,我们提出认知世界模型(CogWM),一种基于大模型的用户认知模拟器,可联合建模用户的BDI/E状态及其对应回应,实现显式的认知轨迹追踪。该模型在15万条用户对话样本上使用Qwen3-14B训练,相比现有用户仿真基线,在回应保真度与认知状态理解方面均表现更优。与六款先进大模型交互实验表明,CogWM能通过认知轨迹实现对对话代理的渐进式对比,揭示不同代理的认知演化模式及与行为结果间的互补关系。
原文摘要 · Abstract (English)
As LLM-based conversational agents advance toward increasingly open-ended and interaction-intensive scenarios, task completion alone provides an incomplete assessment of their effectiveness. The evolution of users' internal states, including beliefs, desires, intentions, and emotions (BDI/E), serves as an intermediate signal connecting agent behaviors with interaction outcomes and reflects how conversational strategies shape users during multi-turn interactions. However, existing evaluation paradigms primarily focus on surface-level responses or final outcomes, providing limited insight into the underlying cognitive processes. This limitation makes it difficult to diagnose why agents succeed or fail and to optimize their interaction strategies. To address this challenge, we propose Cognitive World Model (CogWM), an LLM-based cognitive user model that jointly models users' BDI/E states and corresponding responses, enabling explicit cognitive trajectory tracking. Trained on 150K user-turn samples with Qwen3-14B, CogWM achieves superior performance over existing user simulation baselines in both response fidelity and cognitive state understanding. Interactions with six state-of-the-art LLMs demonstrate that CogWM enables progressive comparison of agents through cognitive trajectories, revealing distinct agent patterns and complementary relationships between cognitive evolution and behavioral outcomes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。