arXiv:2503.20749cs.CL2025-03ACL被引 21

用真实购物数据测试大模型行为模拟能力,发现现有方法准确率仅11.86%。

Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data

论文配图:Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data
图 1 · 摘自论文原文
  • 基于真实用户行为数据,量化评估大模型多轮交互行为模拟效果
  • 纯提示方法准确率仅11.86%,微调后提升至17.26%
  • 适合想提升智能体行为真实性的研究者和产品开发者

近期研究显示,仅通过提示词即可让大模型生成“可信”的人类行为,但现有评估仅关注定性可信度,未检验其在多轮交互中准确复现特定人类行为的能力。本文以购物为例,首次基于大规模真实数据对先进大模型的行为模拟能力进行定量评估。利用包含31,865次在线购物会话、共230,965条用户操作的真实数据,实验表明,基于提示的LLM(DeepSeek-R1、Llama、Claude)在生成人类行为时准确率仅为11.86%,暴露出显著差距。通过简单微调——将真实点击数据与合成推理轨迹结合,模型性能显著提升:微调后的Qwen2.5-7B在行为生成上达17.26%准确率,在最终购买预测上实现33.86% F1分数,分别较提示基线提升5.4%和13.85%。本工作建立了首个严谨的人类行为模拟基准,并为未来应用提供可落地的优化路径。

原文摘要 · Abstract (English)

Recent research shows that LLM Agents can generate ``believable'' human behaviors via prompt-only methods, and such agents have been increasingly adopted in downstream applications. However, existing evaluation of these agents only focuses on qualitative believability (whether human raters think they are accurate), leaving open questions of whether LLM agents can accurately generate step-by-step actions mimicking a particular human's behavior in a multi-turn interaction task. In this work, we take shopping as a case study and present the first large-scale quantitative evaluation of state-of-the-art LLMs' ability to accurately simulate human behavior. Using real-world data from 31,865 online shopping sessions containing 230,965 user actions, our evaluation reveals that prompt-based LLMs (DeepSeek-R1, Llama, Claude) achieve only 11.86% accuracy in generating human actions, highlighting a substantial gap in actual behavioral accuracy. Through experiments, we also showcase that strategies as simple as fine-tuning LLMs on real human click-through data augmented with synthesized reasoning traces can greatly enhance models' performance. The fine-tuned Qwen2.5-7B achieves 17.26% action generation accuracy and 33.86% F1 score on final purchase prediction, representing substantial improvements of 5.4% and 13.85% over prompt-only baselines. This work establishes the first rigorous benchmark for human behavior simulation and provides actionable insights for developing more accurate LLM agents for future downstream applications.

行为模拟大模型评估真实数据购物行为

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。