通过复用分组参数，用更少可训练参数实现更强模型性能。
ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition
- 从预训练权重中提取聚类参数组，共享单一可训练低秩矩阵。
- 在多任务场景下比标准LoRA少54%参数，性能仍更优。
- 适合资源受限但需高精度的LLM微调应用。
LoRA已成为大语言模型快速适应新任务的通用参数高效微调（PEFT）技术。然而，随着模型规模扩大，即使最新变体仍带来显著可训练参数开销。过度降低秩虽能控制开销，却在复杂多任务场景中严重损害性能。我们提出ID-LoRA，一种突破此权衡的新PEFT框架。其核心创新在于从预训练权重矩阵中提取并重用聚类参数组，形成多个共享单一初始化可训练低秩矩阵的低秩组件。该方法在保持模型容量的同时大幅减少可训练参数。我们在五个不同基准上评估：数学推理、代码生成、MMLU、常识问答和安全对齐。ID-LoRA在所有任务上均优于全微调及现有PEFT基线（如LoRA、DoRA、HydraLoRA），且相比标准LoRA最多减少46%可训练参数。在多任务场景中，其在代码与MMLU任务上超越LoRA及其近期变体（如DoRA、HydraLoRA），仅需传统LoRA 54%的可训练参数。
原文摘要 · Abstract (English)
LoRA has become a universal Parameter-Efficient Fine-Tuning (PEFT) technique that equips Large Language Models (LLMs) to adapt quickly to new tasks. However, when these models are scaled up, even the latest LoRA variants still introduce considerable overhead in trainable parameters. Conversely, aggressively lowering the rank to curb this overhead markedly degrades performance in complex multi-task settings. We propose ID-LoRA, a novel PEFT framework that breaks the trade-off. Its core innovation lies in extracting and reusing clustered parameter groups from the pretrained weight matrix. These groups are then used to form multiple low-rank components, all of which share only a single initialized trainable low-rank matrix. This approach cuts the number of trainable parameters while keeping the model's capacity intact. We evaluate ID-LoRA on five diverse benchmarks: Mathematical Reasoning, Code Generation, MMLU, CommonsenseQA, and Safety Alignment. ID-LoRA outperforms both full fine-tuning and existing PEFT baselines (e.g., LoRA, DoRA, HydraLoRA) while using up to 46% fewer trainable parameters than the standard LoRA. In multi-task scenarios, it surpasses LoRA and its recent variants (e.g., DoRA and HydraLoRA) on both Code and MMLU tasks, yet requires only 54% of the trainable parameters demanded by the conventional LoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。