arXiv:2608.02616cs.CLcs.AI2026-08

评测32个跨语言领域隐私信息检测模型,发现小样本微调更有效。

OpenAI Privacy Filter: A Cross-Lingual, Cross-Domain PII Evaluation Across 32 Benchmarks

  • 用XLM-RoBERTa微调,500条数据就超出了OPF零样本表现
  • 零样本下对医疗数据的准确率达0.464,但非拉丁字母上几乎失效
  • 适合快速部署,但需避开文化敏感实体和非拉丁语系

我们首次系统评估了OpenAI隐私过滤器(OPF),这是一个15亿参数的双向隐私信息检测模型,在32个涵盖14种语言和5个领域的基准上进行测试。最实用发现是:在英语合成隐私数据上,仅需约500条标注数据微调XLM-RoBERTa即可超越OPF零样本表现;非英语(如Kiji)需约100条,医疗数据则需约1000条。在小样本下,按类别微调(17类)效率低于二分类标签(n=100时,二分类F1=0.634,类别级仅0.360)。零样本下,OPF在SPY医疗数据集上F1=0.464,AI4Privacy上达0.855,显著优于Presidio和XLM-RoBERTa-large-NER。然而,其性能在非训练分布上急剧下降:通用命名实体识别任务中F1仅为0.04–0.40,阿拉伯语和西里尔字母上分别降至0.04和0.03。错误分析显示,其在结构规则性强的隐私信息(邮箱:0.78,电话:0.76)上表现良好,但在文化差异大的实体(人名:0.40,地址:0.49)上表现差,且普遍偏重召回率(精度0.31–0.54,召回率0.70–0.85)。本文提供使用OPF或微调的决策建议及避坑指南。

原文摘要 · Abstract (English)

We present what is, to our knowledge, the first systematic evaluation of OpenAI's Privacy Filter (OPF), a 1.5B-parameter model that converts an autoregressive language model into a bidirectional PII detector, across 32 benchmarks spanning 14 languages and 5 domains. Our most practically actionable finding is a domain-dependent labeled-data crossover: fine-tuned XLM-RoBERTa surpasses OPF's zero-shot performance with only ~500 labeled examples on English synthetic PII (~100 on non-English Kiji), and ~1000 on synthetic medical PII. Crucially, per-class fine-tuning (17 PII entity types, a subset of OPF's 33) is less data-efficient than binary labels at small n -- at n=100, binary F1=0.634 vs. per-class 0.360. Zero-shot, OPF achieves F1=0.464 on the SPY medical benchmark and F1=0.855 on AI4Privacy, substantially outperforming Presidio and XLM-RoBERTa-large-NER. However, OPF degrades sharply outside its PII training distribution: F1=0.04--0.40 on general NER benchmarks and collapses for non-Latin scripts (Arabic: 0.04, Cyrillic: 0.03). Error analysis reveals OPF excels on structurally regular PII (email: 0.78, phone: 0.76) but struggles with culturally variable entities (person names: 0.40, addresses: 0.49), and is recall-biased across most PII domains (precision 0.31--0.54, recall 0.70--0.85). We provide a decision heuristic for when to use OPF zero-shot, when to fine-tune XLM-RoBERTa, and which language families to avoid.

隐私检测多语言小样本NLP

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。