通过跳过部分层训练，大幅降低医疗NLP联邦学习通信开销。
Federated Learning with Layer Skipping: Efficient Training of Large Language Models for Healthcare NLP
- 只微调预训练大模型的部分层，其余层冻结以减少通信量。
- 通信成本降低约70%，性能仅比集中式训练低2%。
- 适合医疗数据异构且需隐私保护的场景，兼容差分隐私。
联邦学习（FL）可在不共享原始数据的前提下实现跨机构协作训练，有效解决医疗自然语言处理中的隐私问题。然而，在联邦环境下训练大语言模型面临通信开销大和数据异质性等挑战。本文提出层跳过联邦学习（Layer-Skipping FL），即仅在客户端微调预训练大模型的选定层，其余层保持冻结。应用于LLaMA 3.2-1B模型时，该方法使通信成本降低约70%，同时性能与集中式训练相比仅下降2%。我们在i2b2和MIMIC-III数据集上对临床命名实体识别和分类任务进行了评估。实验表明，层跳过联邦学习优于现有基线方法，能有效应对非独立同分布的临床数据分布，并在结合差分隐私时仍具鲁棒性。该方法为医疗NLP中的隐私保护协作学习提供了实用解决方案。
原文摘要 · Abstract (English)
Federated learning (FL) enables collaborative model training across organizations without sharing raw data, addressing crucial privacy concerns in healthcare natural language processing (NLP). However, training large language models (LLMs) in federated settings faces significant challenges, including communication overhead and data heterogeneity. We propose Layer-Skipping Federated Learning, where only selected layers of a pre-trained LLM are fine-tuned across clients while others remain frozen. Applied to LLaMA 3.2-1B, our approach reduces communication costs by approximately 70% while maintaining performance within 2% of centralized training. We evaluate our method on clinical NER and classification tasks using i2b2 and MIMIC-III datasets. Our experiments demonstrate that Layer-Skipping FL outperforms competitive baselines, handles non-IID clinical data distributions effectively, and shows robustness when combined with differential privacy. This approach represents a practical solution for privacy-preserving collaborative learning in healthcare NLP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。