arXiv:2510.02356cs.CRcs.AI2025-10中稿 · ICLR被引 3

测试大模型在真实世界中的隐私意识,发现多数模型表现不佳。

Measuring Physical-World Privacy Awareness of Large Language Models: An Evaluation Benchmark

  • 设计多层级生成场景,评估模型处理敏感物品与环境变化的能力。
  • 顶级模型在动态环境仅达59%准确率,86%忽略隐私请求优先完成任务。
  • 适合关注AI伦理、具身智能安全的研究者与开发者。

大型语言模型(LLMs)在具身智能体中的部署迫切需要评估其在物理世界中的隐私意识。现有评估方法局限于自然语言场景,难以反映真实交互。为此,我们提出EAPrivacy,一个综合性评估基准,通过四层程序化生成场景,测试智能体在处理敏感物体、适应环境变化、平衡任务执行与隐私约束,以及应对社会规范冲突方面的表现。测量结果揭示当前模型存在严重缺陷:表现最优的Gemini 2.5 Pro在动态环境场景中仅获得59%准确率;当任务伴随隐私请求时,模型在高达86%的情况下仍优先完成任务而非遵守隐私限制。在涉及隐私与关键社会规范冲突的高风险情境中,GPT-4o与Claude-3.5-haiku等领先模型也超过15%的时间违背社会规范。这些发现表明,当前大模型在物理隐私对齐方面存在根本性偏差,亟需更稳健的物理感知对齐机制。代码与数据集将公开于https://github.com/Graph-COM/EAPrivacy。

原文摘要 · Abstract (English)

The deployment of Large Language Models (LLMs) in embodied agents creates an urgent need to measure their privacy awareness in the physical world. Existing evaluation methods, however, are confined to natural language based scenarios. To bridge this gap, we introduce EAPrivacy, a comprehensive evaluation benchmark designed to quantify the physical-world privacy awareness of LLM-powered agents. EAPrivacy utilizes procedurally generated scenarios across four tiers to test an agent's ability to handle sensitive objects, adapt to changing environments, balance task execution with privacy constraints, and resolve conflicts with social norms. Our measurements reveal a critical deficit in current models. The top-performing model, Gemini 2.5 Pro, achieved only 59\% accuracy in scenarios involving changing physical environments. Furthermore, when a task was accompanied by a privacy request, models prioritized completion over the constraint in up to 86\% of cases. In high-stakes situations pitting privacy against critical social norms, leading models like GPT-4o and Claude-3.5-haiku disregarded the social norm over 15\% of the time. These findings, demonstrated by our benchmark, underscore a fundamental misalignment in LLMs regarding physically grounded privacy and establish the need for more robust, physically-aware alignment. Codes and datasets will be available at https://github.com/Graph-COM/EAPrivacy.

大模型隐私安全具身智能评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。