让手机助手能用敏感信息但看不到,保护隐私不牺牲功能。
Anonymization-Enhanced Privacy Protection for Mobile GUI Agents: Available but Invisible

- 用识别模型检测敏感内容,替换成带类型标识的占位符。
- 在安卓实验中显著降低隐私泄露,性能损失很小。
- 适合需要高隐私保护的智能助手场景。
移动图形用户界面(GUI)代理通过多模态大语言模型和系统级控制接口,展现出强大的自动化智能手机任务能力。然而,该范式引入了重大隐私风险:代理通常捕获并处理整个屏幕内容,暴露电话号码、地址、消息和财务信息等敏感个人数据。现有防御方法或减少界面暴露、仅混淆无关内容、或依赖用户授权,均无法在保持代理无缝使用的同时保护关键敏感信息。我们提出一种基于匿名化的隐私保护框架,遵循“可用但不可见”原则:敏感信息仍可用于任务执行,但不会直接暴露给云端代理。系统通过一个具备个人信息识别能力的模型检测敏感UI内容,并用确定性、类型保留的占位符(如 PHONE_NUMBER#a1b2c)替换,保留语义类别同时移除识别细节。分层架构包含PII检测器、UI转换器、安全交互代理和隐私网关,确保在用户指令、XML层级和截图中一致匿名化,中介所有代理操作于匿名界面,并在必要时支持局部本地计算。在AndroidLab和PrivScreen基准上的大量实验表明,本框架显著减少隐私泄露,仅造成适度的实用性下降,在现有方法中实现了最佳隐私-实用权衡。代码已公开:https://github.com/one-step-beh1nd/gui_privacy_protection
原文摘要 · Abstract (English)
Mobile Graphical User Interface (GUI) agents have demonstrated strong capabilities in automating complex smartphone tasks by leveraging multimodal large language models (MLLMs) and system-level control interfaces. However, this paradigm introduces significant privacy risks, as agents typically capture and process entire screen contents, thereby exposing sensitive personal data such as phone numbers, addresses, messages, and financial information. Existing defenses either reduce UI exposure, obfuscate only task-irrelevant content, or rely on user authorization, but none can protect task-critical sensitive information while preserving seamless agent usability. We propose an anonymization-based privacy protection framework that enforces the principle of available-but-invisible access to sensitive data: sensitive information remains usable for task execution but is never directly visible to the cloud-based agent. Our system detects sensitive UI content using a PII-aware recognition model and replaces it with deterministic, type-preserving placeholders (e.g., PHONE_NUMBER#a1b2c) that retain semantic categories while removing identifying details. A layered architecture comprising a PII Detector, UI Transformer, Secure Interaction Proxy, and Privacy Gatekeeper ensures consistent anonymization across user instructions, XML hierarchies, and screenshots, mediates all agent actions over anonymized interfaces, and supports narrowly scoped local computations when reasoning over raw values is necessary. Extensive experiments on the AndroidLab and PrivScreen benchmarks show that our framework substantially reduces privacy leakage across multiple models while incurring only modest utility degradation, achieving the best observed privacy-utility trade-off among existing methods. Code available at: https://github.com/one-step-beh1nd/gui_privacy_protection
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。