arXiv:2606.18996cs.CRcs.AI2026-06

测试大模型在完成任务时泄露隐私的倾向,发现越能准确执行任务越容易泄露信息。

TRAP: Benchmark for Task-completion and Resistance to Active Privacy-extraction

论文配图:TRAP: Benchmark for Task-completion and Resistance to Active Privacy-extraction
图 1 · 摘自论文原文
  • 设计新基准TRAP,评估模型任务完成与隐私泄露的平衡
  • 22个模型均存在明显隐私泄露,指令遵循能力越强泄露越高
  • 提出结构化哈希隔离法,可几乎零泄露保高任务成功率

智能体在文档密集型任务中频繁处理敏感信息(如护照号),需精准完成任务但不可暴露私密内容,因无法验证用户身份。这一矛盾导致能力强的模型更易被诱导泄露信息。为此,我们提出任务完成与主动隐私提取抵抗评测基准TRAP:每个场景包含含私密信息的文档、需调用私密字段的任务查询,以及试图以自然语言诱出信息的攻击查询。评估22个前沿开源与闭源模型,发现所有模型家族均存在非平凡泄露,且指令遵循能力与泄露率正相关。现有基于提示的防御手段虽可降低泄露,但显著损害任务准确性;提示优化无法突破此权衡。我们证明:对任意基于Softmax的模型,任何软约束防御(如提示防御)都无法同时实现高任务成功率与零泄露概率。由此提出结构化私密字段隔离方法,在模型前将私密字段替换为哈希键,有效防止泄露同时保持高任务准确率。

原文摘要 · Abstract (English)

Agents are increasingly deployed in document-intensive workflows where sensitive private information is not an edge case but a routine input, e.g., an agent booking a flight needs passport numbers. In such settings, the agent must use private information to complete tasks accurately while never exposing it in its responses, because it cannot verify who is actually at the keyboard. These two obligations are in fundamental tension. A model capable enough to use private information for task completion can, by the same capability, be induced to reveal it. To evaluate the trade-off of task accuracy and privacy leakage, we introduce Task-completion and Resistance to Active Privacy-extraction (TRAP). Each scenario includes a document containing private information, a task query that requires the agent to invoke the correct tool using private fields, and an attack query that attempts to elicit the same information in natural language. Evaluating 22 models spanning frontier proprietary and open-source models at multiple scales, we find that all model families exhibit non-trivial leakage, and that instruction-following ability correlates with leakage rate. Existing prompt-based defenses reduce leakage but at significant cost to task accuracy. Prompt optimization fails to escape this trade-off. We demonstrate that this failure is not incidental. For any softmax-based model, no soft-constraint defense, e.g., prompt-based defenses, can jointly achieve high task success with zero leakage probability. Motivated by this impossibility result, we propose structural private field isolation, which replaces private fields with hash keys before they reach the model. This approach largely prevents leakage while keeping task accuracy.

隐私保护大模型安全任务完成泄露检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。