首次实证研究大模型代理欺骗人类的脆弱性，发现超九成用户难以识别攻击。
"Are You Sure?": An Empirical Study of Human Perception Vulnerability in LLM-Driven Agentic Systems
- 构建高仿真平台HAT-Lab，设计9类真实场景测试人类信任漏洞。
- 仅8.6%参与者察觉欺骗攻击，专家在特定场景更易受骗。
- 有效预警需低干扰、低成本验证，体验式训练可显著提升防范意识。
大型语言模型（LLM）代理正快速成为软件开发、医疗等高风险领域的可信协作者。然而，这种深度信任带来了新攻击面：代理中介欺骗（AMD），即被攻陷的代理反向欺骗其人类用户。尽管已有大量研究关注代理自身威胁，但人类对被劫持代理欺骗的脆弱性尚未被探索。我们通过303名参与者的大规模实证研究，基于自研的高保真科研平台HAT-Lab，涵盖日常与专业领域（如医疗、软件开发、人力资源）的9个精心设计场景，揭示10项关键发现。结果显示，仅有8.6%的参与者识别出AMD攻击，且部分领域专家在特定情境下更易受骗。我们识别出用户六大认知失效模式，并发现风险意识常无法转化为防护行为。防御分析表明，有效的警告应以低验证成本中断工作流。基于HAT-Lab的体验式学习使超过90%感知风险的用户报告防范意识显著提升。本研究为以人为本的代理安全研究提供了实证依据和平台支持。
原文摘要 · Abstract (English)
Large language model (LLM) agents are rapidly becoming trusted copilots in high-stakes domains like software development and healthcare. However, this deepening trust introduces a novel attack surface: Agent-Mediated Deception (AMD), where compromised agents are weaponized against their human users. While extensive research focuses on agent-centric threats, human susceptibility to deception by a compromised agent remains unexplored. We present the first large-scale empirical study with 303 participants to measure human susceptibility to AMD. This is based on HAT-Lab (Human-Agent Trust Laboratory), a high-fidelity research platform we develop, featuring nine carefully crafted scenarios spanning everyday and professional domains (e.g., healthcare, software development, human resources). Our 10 key findings reveal significant vulnerabilities and provide future defense perspectives. Specifically, only 8.6% of participants perceive AMD attacks, while domain experts show increased susceptibility in certain scenarios. We identify six cognitive failure modes in users and find that their risk awareness often fails to translate to protective behavior. The defense analysis reveals that effective warnings should interrupt workflows with low verification costs. With experiential learning based on HAT-Lab, over 90% of users who perceive risks report increased caution against AMD. This work provides empirical evidence and a platform for human-centric agent security research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。