研究联邦学习中分词器对放射科报告隐私泄露的影响
Privacy Leakage in Federated Learning in Radiology Reports: A Comparative Evaluation of Tokenizer-Driven Privacy Risks
- 固定模型架构，对比三种分词器在联邦学习中的隐私风险
- 最大可恢复44%原文句子，且域专用分词器泄漏更严重
- 分词器设计直接影响隐私安全，需结合差分隐私等防护
联邦学习（FL）使多机构可在不共享原始数据的情况下训练临床文本模型，但梯度反演可能从共享模型更新中重建敏感信息。本文量化了放射科报告在FL中的基于梯度的文本重建风险，并在保持模型架构不变的前提下，比较三种分词器的隐私风险。六名客户端使用GPT-2风格Transformer（序列长度32）在公共放射科语料库（368,751份诊断报告、98,206份出院摘要、1,500份MIMIC-CXR自由文本报告）上训练，分别采用GPT-2、RadBERT和LLaMA-2分词器，批量大小为64、128和256。假设存在恶意服务器在分发前修改共享架构，应用分析梯度反演并测量五次运行的重建保真度。整体精确句重建率在31%至44%之间（27个分词器×数据集×批量组合中为30.6%-43.5%）。在批量大小64的出院数据集上，准确率分别为42.1%（GPT-2）、42.3%（RadBERT）和39.4%（LLaMA-2），批量256时降至37.3%、37.2%和34.3%。S-BLEU随批量增大而下降（GPT-2：0.44→0.33；RadBERT：0.48→0.35）。RadBERT重建保真度最高，恢复临床术语占比18.1%（参考词汇表1,440项），高于GPT-2的12.5%和LLaMA-2的9.4%，但无一能阻止泄露。因此，即使在较大批量和领域专用分词器下，报告文本仍可被大量恢复。分词器设计显著影响泄露程度，是隐私相关决策而非仅性能考量；为满足HIPAA和GDPR要求，需结合安全聚合与差分隐私等保护机制。
原文摘要 · Abstract (English)
Federated learning (FL) enables multi-institutional training on clinical text without sharing raw data, but gradient inversion can reconstruct sensitive information from shared model updates. The extent of this leakage for radiology reports, and the role of tokenizer design, remains unclear. We quantify gradient-based text reconstruction in FL and compare privacy risk across three tokenizers with the model architecture held fixed. Six FL clients trained a GPT-2-style transformer (sequence length 32) on public radiology corpora (368,751 diagnostic reports, 98,206 discharge summaries, 1,500 MIMIC-CXR free-text reports) using the GPT-2, RadBERT, and LLaMA-2 tokenizers at batch sizes of 64, 128, and 256. Assuming an active malicious server that modifies the shared architecture before distribution, we applied analytic gradient inversion and measured reconstruction fidelity over five runs. Exact sentence reconstruction ranged from 31% to 44% across tokenizers (30.6-43.5% across the 27 tokenizer x dataset x batch-size cells). At batch size 64 on the Discharge dataset, accuracy was 42.1% (GPT-2), 42.3% (RadBERT), and 39.4% (LLaMA-2), decreasing to 37.3%, 37.2%, and 34.3% at batch size 256. S-BLEU declined as batch size grew (GPT-2: 0.44 to 0.33; RadBERT: 0.48 to 0.35). RadBERT yielded the highest reconstruction fidelity and recovered the most clinical terms (18.1% of a 1,440-term reference vocabulary, vs 12.5% for GPT-2 and 9.4% for LLaMA-2), yet no tokenizer prevented leakage. Substantial portions of report text are therefore recoverable from FL gradients even at larger batch sizes and with domain-specific tokenizers. Tokenizer design influences leakage severity and is a privacy-relevant decision, not only a utility one; safeguards such as secure aggregation and differential privacy are likely necessary to meet HIPAA and GDPR requirements for FL in radiology NLP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。