FedDPG在联邦学习中动态生成适配输入的提示,提升性能并降低通信开销。
FedDPG: An Adaptive Yet Efficient Prompt-tuning Approach in Federated Learning Settings
- 引入动态提示生成网络,根据输入自适应生成提示。
- 在三个NLP数据集上超越现有方法,计算时间减少40%以上。
- 适合资源受限且注重隐私保护的联邦学习场景。
预训练语言模型(PLMs)在多种自然语言任务中表现优异。但传统微调方法计算开销大。提示调优(Prompt-tuning)通过在输入前添加少量可训练参数,在冻结PLM参数的前提下实现高效调优。然而,该方法对所有输入使用固定提示,灵活性不足。联邦学习(FL)近年受到关注,以应对日益增长的数据隐私担忧,但客户端的通信与计算限制仍待解决。为此,本文提出联邦动态提示生成器(FedDPG),引入动态提示生成网络,根据输入生成上下文感知的提示,兼顾灵活性与数据隐私。在三个NLP基准数据集上的实验表明,FedDPG在全局模型性能上优于当前最先进的参数高效微调方法,且计算时间显著降低,需传输的参数量减少约60%。
原文摘要 · Abstract (English)
Pre-trained Language Models (PLMs) have demonstrated impressive performance in various NLP tasks. However, traditional fine-tuning methods for leveraging PLMs for downstream tasks entail significant computational overhead. Prompt-tuning has emerged as an efficient alternative that involves prepending a limited number of parameters to the input sequence and only updating them while the PLM's parameters are frozen. However, this technique's prompts remain fixed for all inputs, reducing the model's flexibility. The Federated Learning (FL) technique has gained attention in recent years to address the growing concerns around data privacy. However, challenges such as communication and computation limitations of clients still need to be addressed. To mitigate these challenges, this paper introduces the Federated Dynamic Prompt Generator (FedDPG), which incorporates a dynamic prompt generator network to generate context-aware prompts based on the given input, ensuring flexibility and adaptability while prioritising data privacy in federated learning settings. Our experiments on three NLP benchmark datasets showcase that FedDPG outperforms the state-of-the-art parameter-efficient fine-tuning methods in terms of global model performance, and has significantly reduced the calculation time and the number of parameters to be sent through the FL network.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。