首个记录用户与AI对话中真实想法的大规模数据集。
ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions
- 收集用户在20个大模型上的思考片段，匹配实际对话
- 思想与对话内容语义不同，大模型难以推断
- 可用于预测用户行为和训练个性化助手
对话式AI已覆盖数十亿用户，但现有数据集仅记录用户所说，未捕捉其内心所思。我们提出ThoughtTrace，首个大规模真实世界多轮人机对话与用户自述想法的配对数据集：包括用户发送提示的原因及对回复的反应。该数据集涵盖1,058名用户、2,155次对话、17,058轮交互和10,174条思想标注，覆盖20个语言模型。分析显示，ThoughtTrace能捕捉长周期、话题多样化的互动；思想在语义上区别于对话内容，当前前沿大模型难以从上下文中推断，内容多样且与对话阶段相关。我们进一步验证了思想的下游价值：第一，作为推理时上下文可提升用户行为预测性能；第二，通过思想引导的重写可提供细粒度对齐信号，用于训练个性化助手。ThoughtTrace将用户思想确立为研究人机交互认知动态的新数据模态，为构建更懂用户隐性目标、偏好与需求的智能助手奠定基础。
原文摘要 · Abstract (English)
Conversational AI has now reached billions of users, yet existing datasets capture only what people say, not what they think. We introduce ThoughtTrace, the first large-scale dataset that pairs real-world multi-turn human--AI conversations with users' self-reported thoughts: their reasons for sending prompts and reactions to assistant responses. ThoughtTrace comprises 1,058 users, 2,155 conversations, 17,058 turns, and 10,174 thought annotations collected across 20 language models. Our analysis shows that ThoughtTrace captures long-horizon, topically diverse interactions, and that thoughts are semantically distinct from messages, difficult for frontier LLMs to infer from context, diverse in content, and tied to conversation stages. We further demonstrate the utility of thoughts for downstream modeling. First, thoughts improve user-behavior prediction as inference-time context. Second, thought-guided rewrites provide fine-grained alignment signals for training personalized assistants. Together, ThoughtTrace establishes user thoughts as a new data modality for studying the cognitive dynamics behind human--AI interaction and provides a foundation for building assistants that better understand and adapt to users' latent goals, preferences, and needs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。