提出DeCAF算法，让低秩适配在去中心化场景下更快更稳地收敛。
DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models
- 用截断奇异值分解重构低秩矩阵，解决去中心化训练中的共识干扰问题。
- 理论证明其收敛速度与去中心化SGD相当，且随秩增大误差趋近于零。
- 在视觉语言任务中优于本地训练和联邦学习，适用于非独立同分布数据。
低秩适配（LoRA）是训练视觉-语言模型（VLMs）和大语言模型（LLMs）的一种高效、计算可行的微调方法。它通过冻结预训练模型权重并注入可训练的低秩矩阵，实现在边缘设备上对基础模型的有效学习。然而，去中心化环境下的LoRA仍研究不足，尤其缺乏平滑性保证和模型共识干扰的理论分析。本文通过确保梯度平滑性，将去中心化LoRA（DLoRA）的收敛速度提升至与去中心化SGD相当。同时提出DeCAF，一种结合DLoRA与基于截断奇异值分解（TSVD）的矩阵分解的新算法，以解决共识干扰问题。理论分析表明，TSVD的近似误差有界，且随着秩增加，DLoRA与DeCAF之间的共识差异趋于消失，从而实现一致的收敛率。大量实验表明，在多种视觉-语言任务中，该算法在独立同分布（IID）和非独立同分布（non-IID）数据分布下均优于本地训练和现有联邦学习方法。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) has emerged as one of the most effective, computationally tractable fine-tuning approaches for training Vision-Language Models (VLMs) and Large Language Models (LLMs). LoRA accomplishes this by freezing the pre-trained model weights and injecting trainable low-rank matrices, allowing for efficient learning of these foundation models even on edge devices. However, LoRA in decentralized settings still remains under explored, particularly for the theoretical underpinnings due to the lack of smoothness guarantee and model consensus interference (defined formally below). This work improves the convergence rate of decentralized LoRA (DLoRA) to match the rate of decentralized SGD by ensuring gradient smoothness. We also introduce DeCAF, a novel algorithm integrating DLoRA with truncated singular value decomposition (TSVD)-based matrix factorization to resolve consensus interference. Theoretical analysis shows TSVD's approximation error is bounded and consensus differences between DLoRA and DeCAF vanish as rank increases, yielding DeCAF's matching convergence rate. Extensive experiments across vision/language tasks demonstrate our algorithms outperform local training and rivals federated learning under both IID and non-IID data distributions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。