动态调整专家数量与秩，让大模型微调更高效。
Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning
- 分层配置专家数量和秩，匹配不同层的表达复杂度。
- 在多个任务上准确率更高，参数量减少30%以上。
- 适合追求高效微调的大模型应用开发者。
大型语言模型（LLMs）在各类任务中表现出色，其参数规模持续增长。参数高效微调（PEFT）方法如低秩适应（LoRA）通过显著减少可训练参数，缓解了微调难题。近期研究将LoRA与专家混合（MoE）架构结合，利用多个适配器专家和门控机制提升微调性能。然而，现有方法主要关注每层适配器专家的分配以优化参数量，忽略了适配器秩这一关键因素。为此，我们提出层级化专家分配与秩配置方案HILO，动态调整各层适配器专家的数量与秩，以匹配模型层在适配器粒度下的表示复杂度。在多个基准任务上的实验表明，HILO在保持更低可训练参数的同时，准确率优于现有方法，为大模型微调提供了高效且实用的解决方案。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated remarkable success across various tasks, accompanied by a continuous increase in their parameter size. Parameter-efficient fine-tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), address the challenges of fine-tuning LLMs by significantly reducing the number of trainable parameters. Recent studies have integrated LoRA with Mixture of Experts (MoE) architectures, leveraging multiple adapter experts and gating mechanisms to further improve fine-tuning performance. However, existing approaches primarily focus on adjusting the allocations of adapter experts per layer to optimize the introduced trainable parameter size, while neglecting a critical factor of adapters' rank. To this end, we propose a hierarchical scheme for expert allocation and rank configuration, HILO, which dynamically adjusts the number and rank of adapter experts across layers, matching the varying representational complexity of model layers in adapter-granularity. Extensive experiments on multiple benchmark tasks demonstrate that HILO outperforms existing methods in accuracy while introducing fewer trainable parameters, providing an efficient and practical solution for fine-tuning LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。