从搜索行为日志推断用户思维过程，提升搜索系统评估精度。
Beyond the Click: A Framework for Inferring Cognitive Traces in Search
- 基于信息觅食理论构建多智能体LLM框架，推断用户认知状态。
- 在MovieLens上，认知模型使F1提升6.6%，优于行为基线和随机标签。
- 适用于需要理解用户心理的搜索系统评测，支持开源复现。
用户模拟器对搜索系统评估至关重要，但通常只复现用户行为而未建模其思维过程。大规模交互日志记录了用户操作，却缺乏其潜在思考或情绪（如困惑、满意）的信息。本文提出一种从行为日志推断认知痕迹的框架，采用基于信息觅食理论（IFT）的多智能体大语言模型，并经人类专家验证。我们在AOL、Stack Overflow和MovieLens三个公开数据集上标注了超过53万条认知标签，覆盖5万个会话。跨数据集评估显示，在行为特征较弱时，认知标签提供最强信号：在MovieLens上，认知模型相比行为基线提升F1达6.6%，比随机标签控制组高1.8%；而在点击模式高度可预测的AOL上，改进接近零。我们已将标注数据集发布于HuggingFace，开放源码工具及全部实验代码，以支持未来认知感知型用户模拟研究。
原文摘要 · Abstract (English)
User simulators are essential for evaluating search systems, but they primarily reproduce user actions without modeling the underlying thought process. Large-scale interaction logs record what users do, but not what they might be thinking or feeling, such as confusion or satisfaction. We present a framework for inferring cognitive traces from behavioral logs. Our method uses a multi-agent LLM system grounded in Information Foraging Theory (IFT) and validated by human experts. We annotate three public datasets (AOL, Stack Overflow, and MovieLens), producing over 530,000 cognitive labels across 50,000 sessions. A cross-dataset evaluation with a shuffled-label control reveals that cognitive labels provide the strongest signal where behavioral features are weakest: on MovieLens, the cognitive model improves F1 by up to 6.6% over the behavioral baseline and 1.8% above the shuffled control, while on AOL, where click patterns are highly predictive, improvements are near zero. We release the annotation collection on HuggingFace, an open-source annotation tool, and all experimental code to support future work on cognitively aware user simulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。