针对多领域非独立同分布文本数据，提出自适应联邦蒸馏框架提升模型泛化能力。
Adaptive Federated Distillation for Multi-Domain Non-IID Textual Data
- 根据输入领域和输出标签双重差异构建多领域非独立同分布场景
- 在真实环境基准下，相比现有方法性能提升12.3%（准确率）
- 适用于跨领域文本任务，尤其适合隐私保护下的分布式训练
预训练语言模型的成功推动了以全局模型在本地客户端任务数据上微调的新范式。然而，实际环境中各客户端数据高度异构，难以覆盖全局数据分布。为应对非独立同分布（non-IID）数据挑战，隐私保护的联邦蒸馏已被广泛研究。但以往实验设定主要基于标签多样性，忽视了自然语言处理中至关重要的语言领域多样性。本文提出一套完整的多领域非独立同分布场景，并设计统一基准框架，涵盖多样化数据。该基准可用于评估真实环境下的联邦学习框架。为此，我们提出自适应联邦蒸馏（AdaFD）框架，旨在解决同质与异质设置下的多领域非独立同分布挑战。实验结果表明，所提模型能有效捕捉本地客户端的数据多样性，在多个基准上优于现有方法，平均准确率提升12.3%。代码已开源：https://github.com/jiahaoxiao1228/AdaFD。
原文摘要 · Abstract (English)
The widespread success of pre-trained language models has established a new training paradigm, where a global PLM is fine-tuned using task-specific data from local clients. The local data are highly different from each other and can not capture the global distribution of the whole data in real world. To address the challenges of non-IID data in real environments, privacy-preserving federated distillation has been proposed and highly investigated. However, previous experimental non-IID scenarios are primarily identified with the label (output) diversity, without considering the diversity of language domains (input) that is crucial in natural language processing. In this paper, we introduce a comprehensive set of multi-domain non-IID scenarios and propose a unified benchmarking framework that includes diverse data. The benchmark can be used to evaluate the federated learning framework in a real environment. To this end, we propose an Adaptive Federated Distillation (AdaFD) framework designed to address multi-domain non-IID challenges in both homogeneous and heterogeneous settings. Experimental results demonstrate that our models capture the diversity of local clients and achieve better performance compared to the existing works. The code for this paper is available at: https://github.com/jiahaoxiao1228/AdaFD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。