构建统一中文隐私信息检测基准，解决数据碎片化难题
PIIBench: A Unified Multi-Source Benchmark Corpus for Personally Identifiable Information Detection
- 整合10个数据集，统一标注格式为BIO标准
- 含335万实体提及，48类隐私信息，跨领域覆盖全面
- 揭示现有模型性能极低，验证评测难度显著提升
我们提出PIIBench，一个统一的自然语言文本中个人身份信息（PII）检测基准语料库。现有PII检测资源分散于不同领域的语料库，标注方案互不兼容，阻碍系统间系统性比较。我们整合了涵盖合成数据、多语言命名实体识别基准及金融领域标注文本的10个公开数据集，形成包含2,369,883个标注序列和335万实体提及的语料库，覆盖48种标准PII实体类型。开发了一套规范化的处理流程，将80余种源标签变体映射至统一的BIO标注体系，基于频率抑制近似不存在的实体类型，并生成保留来源分布的80/10/10分层训练/验证/测试集。为建立基线难度，评估了八种已发表系统：规则引擎（Microsoft Presidio）、通用NER模型（spaCy、BERT-base NER、XLM-RoBERTa NER、SpanMarker mBERT、SpanMarker BERT）、专用PII模型（Piiranha DeBERTa）及金融领域专家模型（XtremeDistil FiNER）。所有系统在片段级F1均低于0.14，最优系统（Presidio，F1=0.1385）对多数实体类型仍无法召回。结果直接量化了领域孤岛问题，证明PIIBench比任何单一来源的PII数据集更具挑战性。数据构建流程与评测代码已开源。
原文摘要 · Abstract (English)
We present PIIBench, a unified benchmark corpus for Personally Identifiable Information (PII) detection in natural language text. Existing resources for PII detection are fragmented across domain-specific corpora with mutually incompatible annotation schemes, preventing systematic comparison of detection systems. We consolidate ten publicly available datasets spanning synthetic PII corpora, multilingual Named Entity Recognition (NER) benchmarks, and financial domain annotated text, yielding a corpus of 2,369,883 annotated sequences and 3.35 million entity mentions across 48 canonical PII entity types. We develop a principled normalization pipeline that maps 80+ source-specific label variants to a standardized BIO tagging scheme, applies frequency-based suppression of near absent entity types, and produces stratified 80/10/10 train/validation/test splits preserving source distribution. To establish baseline difficulty, we evaluate eight published systems spanning rule-based engines (Microsoft Presidio), general purpose NER models (spaCy, BERT-base NER, XLM-RoBERTa NER, SpanMarker mBERT, SpanMarker BERT), a PII-specific model (Piiranha DeBERTa), and a financial NER specialist (XtremeDistil FiNER). All systems achieve span-level F1 below 0.14, with the best system (Presidio, F1=0.1385) still producing zero recall on most entity types. These results directly quantify the domain-silo problem and demonstrate that PIIBench presents a substantially harder and more comprehensive evaluation challenge than any existing single source PII dataset. The dataset construction pipeline and benchmark evaluation code are publicly available at https://github.com/pritesh-2711/pii-bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。