通过聚类与张量分解，减少多任务微调中的干扰，提升模型表现。
Tensorized Clustered LoRA Merging for Multi-Task Interference
- 按输入相似性聚类数据，为每类训练专用LoRA适配器。
- 采用联合CP分解分离任务特有与共享参数，降低干扰。
- 在推理、问答、编程任务上比基线提升2.3%，适合多任务场景。
尽管大规模语言模型的密集架构已取得成功，但LoRA适配器通过微调小型任务特定模块并合并到基础模型中，提供了高效解决方案。然而，在多任务设置下，由异构来源训练的LoRA适配器频繁引发任务干扰，导致下游性能下降。为此，我们提出张量化聚类LoRA（TC-LoRA）方法，旨在从文本层面和参数层面解决任务干扰问题。在文本层面，我们在嵌入空间中对训练样本进行聚类，以捕捉输入格式相似性，并为每个聚类训练专用的LoRA适配器；在参数层面，引入联合正交分解（CP）来解耦不同LoRA适配器间的任务特有与共享因子。该联合分解保留关键知识的同时，有效降低跨任务干扰。在跨域零样本及技能组合任务（包括推理、问答、编程）上的大量实验表明，相比强基线（基于SVD的方法），TC-LoRA在Phi-3上实现+1.4%准确率提升，在Mistral-7B上达到+2.3%的提升，验证了其在大模型适应中的有效性。
原文摘要 · Abstract (English)
Despite the success of the monolithic dense paradigm of large language models (LLMs), the LoRA adapters offer an efficient solution by fine-tuning small task-specific modules and merging them with the base model. However, in multi-task settings, merging LoRA adapters trained on heterogeneous sources frequently causes \textit{task interference}, degrading downstream performance. To address this, we propose a tensorized clustered LoRA (TC-LoRA) library targeting to address the task interference at the \textit{text-level} and \textit{parameter-level}. At the \textit{text-level}, we cluster the training samples in the embedding space to capture input-format similarities, then train a specialized LoRA adapter for each cluster. At the \textit{parameter-level}, we introduce a joint Canonical Polyadic (CP) decomposition that disentangles task-specific and shared factors across LoRA adapters. This joint factorization preserves essential knowledge while reducing cross-task interference. Extensive experiments on out-of-domain zero-shot and skill-composition tasks-including reasoning, question answering, and coding. Compared to strong SVD-based baselines, TC-LoRA achieves +1.4\% accuracy on Phi-3 and +2.3\% on Mistral-7B (+2.3\%), demonstrating the effectiveness of TC-LoRA in LLM adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。