构建文档级隐私泄露风险评估基准，揭示红印后仍存高危泄露隐患
LeakageBench: Document-Level Leakage Risk for Redacting Personally Identifiable Information in Document Images
- 提出覆盖1.2万条合规标注的文档图像泄露风险评测集
- 实测多数页面红印后仍存96.8%泄露概率，检测精度不等于安全
- 适合隐私保护、文档自动化处理及AI模型安全研究者使用
现实中的个人身份信息（PII）红印多针对文档图像——扫描件、截图和PDF渲染图——其中光学字符识别错误、版面结构和视觉噪声决定敏感信息是否真正被清除。现有PII基准多聚焦文本层面，未衡量文档级红印风险：只要遗漏一个标识符，整页即不安全。我们提出LeakageBench，一个包含500张文档图像的挑战数据集，含11,954个符合GDPR标准的PII标注，涵盖直接标识符、链接键和上下文再识别表面。我们评估通用OCR流程、商业与任务适配型OCR依赖检测器，以及OCR-free视觉语言模型，采用实体级F1、组级泄露和文档级泄露指标。代码解释器将GPT-5.5定位F1从0.090提升至0.249，但关键页面级泄露率仍达0.968。结果表明，更强检测与工具辅助可改善定位，却无法让多数页面真正安全。LeakageBench为高召回、空间精准的文档图像PII红印提供诊断性基准。
原文摘要 · Abstract (English)
Real-world personally identifiable information (PII) redaction often operates on document images---scans, screenshots, and PDF renderings---where OCR errors, layout structure, and visual noise determine whether sensitive information is actually removed. Existing PII benchmarks are mostly text-centric and do not measure document-level redaction risk: a page remains unsafe if even one identifier is missed. We introduce LeakageBench, a challenge set of 500 document images with 11,954 GDPR-aligned PII annotations spanning direct identifiers, linkage keys, and contextual re-identification surfaces. We evaluate generic OCR pipelines, commercial and task-adapted OCR-dependent detectors, and OCR-free vision-language models using entity-level F1, group-wise leakage, and document-level leakage metrics. Code Interpreter raises GPT-5.5 localization F1 from 0.090 to 0.249, but critical page-level leakage remains 0.968. These results show that stronger detection and tool assistance improve localization without making most pages safe for release. LeakageBench provides a diagnostic benchmark for high-recall, spatially grounded PII redaction in document images.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。