arXiv:2411.06581cs.LGcs.AI2024-11中稿 · the 2025 IEEE Inte…被引 1

HAFLQ通过自适应量化与参数裁剪,提升异构环境下的大模型联邦微调效率。

HAFLQ: Heterogeneous Adaptive Federated LoRA Fine-tuned LLM with Quantization

  • 基于重要性动态量化模型层,降低内存与计算开销。
  • 通信成本降49%,精度提升50%,收敛速度更快。
  • 适合资源受限、设备异构的隐私保护场景使用。

联邦微调预训练大语言模型可在保护隐私的前提下实现跨数据集的任务适配,但面临高算力需求、客户端资源异构、带宽限制及全局聚合效率低等挑战。为此,我们提出HAFLQ框架,通过显著性驱动的自适应量化策略,根据变换器模块的重要性实施分块量化;针对异构算力,采用基于重要性的参数截断与冻结机制;为缓解通信瓶颈,设计重要性感知的带宽自适应量化方法,动态调整参数精度;并提出自适应秩1矩阵级聚合策略,仅融合更新的秩1矩阵以防止信息稀释,加速收敛。在文本分类任务上的实验表明,相比基线方法,HAFLQ降低31%内存占用,通信成本减少49%,准确率提升50%,且收敛更快。

原文摘要 · Abstract (English)

Federated fine-tuning of pre-trained Large Language Models (LLMs) enables task-specific adaptation across diverse datasets while preserving privacy. However, challenges such as high computational and memory demands, heterogeneous client resources, bandwidth constraints, and ineffective global aggregation hinder its efficiency. To address these issues, we propose HAFLQ (Heterogeneous Adaptive Federated Low-Rank Adaptation Fine-tuned LLM with Quantization), a novel framework for efficient and scalable federated fine-tuning of LLMs in heterogeneous environments. To reduce memory and computation demands, we propose a salience-driven adaptive LLM quantization framework that evaluates the importance of transformer blocks using a salience metric and applies adaptive block-wise quantization accordingly. To handle heterogeneous computational capabilities, we propose an importance-based parameter truncation and freezing scheme. To address communication bottlenecks, we propose an importance-aware bandwidth-adaptive quantization method, which dynamically adjusts parameter precision based on importance and bandwidth constraints. To improve global model aggregation, we propose an adaptive rank-1 matrix-level aggregation strategy, which prevents information dilution and accelerates convergence by aggregating only updated rank-1 matrices from clients. Experimental results on the text classification task demonstrate that HAFLQ reduces memory usage by 31%, lowers communication cost by 49%, improves accuracy by 50%, and achieves faster convergence compared to the baseline method.

联邦学习大模型微调量化异构计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。