让问答系统智能识别哪些隐私信息该保留,提升回答质量。
CAPID: Context-Aware PII Detection for Question-Answering Systems
- 用小模型细调,判断隐私信息是否与问题相关
- 在保持隐私前提下,下游问答准确率提升23%以上
- 自动生成带上下文的合成数据,适合安全场景
在问答系统中检测用户查询中的个人身份信息(PII)对保护隐私至关重要。现有方法通常直接删除所有PII,忽视其可能与问题相关的上下文价值,导致回答质量下降。虽然大语言模型(LLMs)可辅助判断相关性,但因闭源且缺乏隐私保障,不适合处理敏感数据。为此,我们提出CAPID:一种本地部署的小语言模型(SLM)微调方案,在PII传入LLM前进行过滤。现有数据集无法充分捕捉PII的上下文相关性,因此我们设计了一种基于LLMs的合成数据生成管道,构建涵盖多种PII类型和相关性层级的多样化、领域丰富的数据集。利用该数据集,我们微调一个SLM以检测PII片段、分类类型并估计上下文相关性。实验表明,使用微调后的SLM进行相关性感知的PII检测,在片段、相关性和类型准确性上显著优于基线,同时在匿名化后仍保持更高的下游任务效用(提升23%以上)。
原文摘要 · Abstract (English)
Detecting personally identifiable information (PII) in user queries is critical for ensuring privacy in question-answering systems. Current approaches mainly redact all PII, disregarding the fact that some of them may be contextually relevant to the user's question, resulting in a degradation of response quality. Large language models (LLMs) might be able to help determine which PII are relevant, but due to their closed source nature and lack of privacy guarantees, they are unsuitable for sensitive data processing. To achieve privacy-preserving PII detection, we propose CAPID, a practical approach that fine-tunes a locally owned small language model (SLM) that filters sensitive information before it is passed to LLMs for QA. However, existing datasets do not capture the context-dependent relevance of PII needed to train such a model effectively. To fill this gap, we propose a synthetic data generation pipeline that leverages LLMs to produce a diverse, domain-rich dataset spanning multiple PII types and relevance levels. Using this dataset, we fine-tune an SLM to detect PII spans, classify their types, and estimate contextual relevance. Our experiments show that relevance-aware PII detection with a fine-tuned SLM substantially outperforms existing baselines in span, relevance and type accuracy while preserving significantly higher downstream utility under anonymization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。