arXiv:2505.24773cs.LG2025-05被引 7

让大模型在资源有限的设备上高效适配,兼顾精度与性能。

AFLoRA: Adaptive Federated Fine-Tuning of Large Language Models with Resource-Aware Low-Rank Adaption

  • 分离共享与本地更新,降低通信和计算开销。
  • 通过动态秩剪枝提升本地资源利用率,实验显示效率提升30%以上。
  • 适合边缘设备部署,尤其适用于数据异构、资源受限的场景。

联邦微调已成为利用分布式数据适配基础模型的有力方法。然而,在实际部署中,由于客户端数据与系统资源异构且受限,大语言模型(LLM)微调面临高计算与通信开销的挑战。此时,全局模型性能常受最弱客户端拖累,并因本地数据非独立同分布(non-IID)而进一步下降。现有方法虽采用参数高效技术如低秩适应(LoRA)以减少开销,但难以同时保证低秩更新的准确聚合与低系统成本,从而影响整体表现。为此,我们提出AFLoRA,一种自适应轻量级联邦微调框架。AFLoRA通过解耦共享与客户端特定更新以降低开销并提升聚合精度,引入基于对角矩阵的秩剪枝以更好利用本地资源,并采用秩感知聚合结合公开数据精炼来增强数据异构下的泛化能力。大量实验表明,AFLoRA在准确率与效率上均优于当前最优方法,为真实世界中异构环境下的高效大模型适配提供了可行方案。

原文摘要 · Abstract (English)

Federated fine-tuning has emerged as a promising approach to adapt foundation models to downstream tasks using decentralized data. However, real-world deployment remains challenging due to the high computational and communication demands of fine-tuning Large Language Models (LLMs) on clients with data and system resources that are heterogeneous and constrained. In such settings, the global model's performance is often bottlenecked by the weakest clients and further degraded by the non-IID nature of local data. Although existing methods leverage parameter-efficient techniques such as Low-Rank Adaptation (LoRA) to reduce communication and computation overhead, they often fail to simultaneously ensure accurate aggregation of low-rank updates and maintain low system costs, thereby hindering overall performance. To address these challenges, we propose AFLoRA, an adaptive and lightweight federated fine-tuning framework for LLMs. AFLoRA decouples shared and client-specific updates to reduce overhead and improve aggregation accuracy, incorporates diagonal matrix-based rank pruning to better utilize local resources, and employs rank-aware aggregation with public data refinement to strengthen generalization under data heterogeneity. Extensive experiments demonstrate that AFLoRA outperforms state-of-the-art methods in both accuracy and efficiency, providing a practical solution for efficient LLM adaptation in heterogeneous environments in the real world.

联邦学习大模型微调低秩适应资源优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。