通过分布式正交适配器提升大模型微调表达能力，性能显著优于传统方法。
HD-PiSSA: High-Rank Distributed Orthogonal Adaptation
- 在多设备上分配不同主成分的正交适配器，扩展更新方向范围。
- 8卡并行下有效更新秩超数据并行LoRA或PiSSA的16倍以上。
- 在数学、代码生成等任务中平均提升超10点，适合复杂下游任务。
现有高效参数微调（PEFT）方法如LoRA和PiSSA将模型更新限制在低秩子空间，制约了表达能力，导致复杂任务表现不佳。为此，我们提出高秩分布式正交适配（HD-PiSSA），在不同设备上初始化正交适配器，并在微调时对权重W集体聚合其增量更新。与所有设备保持相同适配器的数据并行LoRA或PiSSA不同，HD-PiSSA将预训练权重的不同主成分分配给每张GPU，显著扩展了更新方向范围。在8张GPU上使用相同每设备适配器秩时，其有效更新秩超过数据并行LoRA或PiSSA的16倍以上。我们在多个挑战性下游任务中评估该方法，包括数学、代码生成和多任务学习。在多任务设置中，HD-PiSSA在12个基准上平均比LoRA提升10.0绝对点（14.63%），比PiSSA提升4.98点（6.60%），验证了其因额外优化灵活性带来的优势。
原文摘要 · Abstract (English)
Existing parameter-efficient fine-tuning (PEFT) methods for large language models (LLMs), such as LoRA and PiSSA, constrain model updates to low-rank subspaces, limiting their expressiveness and leading to suboptimal performance on complex tasks. To address this, we introduce High-rank Distributed PiSSA (HD-PiSSA), a distributed PEFT approach that initializes orthogonal adapters across different devices and aggregates their delta updates collectively on W for fine-tuning. Unlike Data Parallel LoRA or PiSSA, which maintain identical adapters across all devices, HD-PiSSA assigns different principal components of the pre-trained weights to each GPU, significantly expanding the range of update directions. This results in over 16x higher effective updated ranks than data-parallel LoRA or PiSSA when fine-tuning on 8 GPUs with the same per-device adapter rank. Empirically, we evaluate HD-PiSSA across various challenging downstream tasks, including mathematics, code generation, and multi-task learning. In the multi-task setting, HD-PiSSA achieves average gains of 10.0 absolute points (14.63%) over LoRA and 4.98 points (6.60%) over PiSSA across 12 benchmarks, demonstrating its benefits from the extra optimization flexibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。