只微调关键注意力层，提升医疗文本分类的隐私与效率
Selective Attention Federated Learning: Improving Privacy and Efficiency for Clinical Text Classification
- 根据注意力模式动态选择重要层进行微调
- 通信量减少，差分隐私防护能力更强
- 适合医疗等隐私敏感场景的高效模型训练
联邦学习（FL）在训练大语言模型时面临通信开销大和模型隐私保护难的问题，尤其在医疗应用中更为突出。为此，我们提出选择性注意力联邦学习（SAFL），通过分析注意力模式识别关键变换层，仅对这些层进行动态微调。该方法显著降低通信带宽需求，并增强差分隐私的鲁棒性。在临床自然语言处理基准数据集i2b2临床概念提取和MIMIC-III出院总结上的实验表明，SAFL在保持与集中式模型相当性能的同时，大幅提升了通信效率和隐私保护能力。
原文摘要 · Abstract (English)
Federated Learning (FL) faces major challenges regarding communication overhead and model privacy when training large language models (LLMs), especially in healthcare applications. To address these, we introduce Selective Attention Federated Learning (SAFL), a novel approach that dynamically fine-tunes only those transformer layers identified as attention-critical. By employing attention patterns to determine layer importance, SAFL significantly reduces communication bandwidth and enhances differential privacy resilience. Evaluations on clinical NLP benchmarks (i2b2 Clinical Concept Extraction and MIMIC-III discharge summaries) demonstrate that SAFL achieves competitive performance with centralized models while substantially improving communication efficiency and privacy preservation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。