用大模型实时识别人机协作中的恶意行为，无需特定任务信息。
Detection of adversarial intent in Human-AI teams using LLMs
- 利用对话记录分析，让大模型充当人机团队的防御监督者。
- 在25轮真实交互中，模型可实时检测恶意行为且不依赖任务知识。
- 适合需要安全防护的智能协作场景，如决策支持系统。
大型语言模型（LLMs）正越来越多地被部署在人机协作团队中，作为信息检索、编程和决策辅助等复杂任务的支持代理。尽管其自主性和上下文理解能力使其具有价值，但也使其面临数据污染、提示注入和提示工程等多种攻击。攻击者可通过这些途径操纵模型输出有害信息，进而误导人类成员做出错误决策。现有研究多关注模型作为攻击目标或攻击方，本文则探讨其作为混合人机团队中防御监督者的潜力。基于一个包含25轮真实人机团队对话与决策的多参与方数据集，我们提出从交互痕迹中检测恶意行为的问题。实验表明，仅依赖交互记录，大模型即可实时识别恶意行为，且无需特定任务信息，显示出任务无关防御的潜力。此外，这类恶意行为难以通过简单启发式规则发现，进一步说明引入大模型防御者可增强人类团队对特定攻击类别的鲁棒性。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly deployed in human-AI teams as support agents for complex tasks such as information retrieval, programming, and decision-making assistance. While these agents' autonomy and contextual knowledge enables them to be useful, it also exposes them to a broad range of attacks, including data poisoning, prompt injection, and even prompt engineering. Through these attack vectors, malicious actors can manipulate an LLM agent to provide harmful information, potentially manipulating human agents to make harmful decisions. While prior work has focused on LLMs as attack targets or adversarial actors, this paper studies their potential role as defensive supervisors within mixed human-AI teams. Using a dataset consisting of multi-party conversations and decisions for a real human-AI team over a 25 round horizon, we formulate the problem of malicious behavior detection from interaction traces. We find that LLMs are capable of identifying malicious behavior in real-time, and without task-specific information, indicating the potential for task-agnostic defense. Moreover, we find that the malicious behavior of interest is not easily identified using simple heuristics, further suggesting the introduction of LLM defenders could render human teams more robust to certain classes of attack.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。