解决多任务微调中路由不一致问题，让模型更稳定高效。
CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning
- 不用路由模块，通过一致性对齐共享低秩空间
- 在多个数据集上优于现有最先进方法，提升稳定性
- 适合需要稳定多任务适配的LLM应用
尽管多任务学习对大语言模型适配多样化领域至关重要，但现有的基于LoRA的方法依赖复杂的路由机制来划分任务特定知识。本文揭示，这类路由设计易产生训练-推理不一致，在分布偏移下随机路由决策会损害推理稳定性。基于二阶泰勒分析发现路由方差带来的不稳定性，我们提出一致性驱动的低秩适配（CD-LoRA）。通过完全去除路由模块，CD-LoRA在共享低秩空间中采用一致性对齐机制，强制跨任务表征一致，形成鲁棒、任务无关的特征表示，无需显式分区开销。大量实验表明，CD-LoRA持续优于当前最先进的多适配器基线，提供一种更简单、无路由、更稳定的多任务参数高效微调方案。
原文摘要 · Abstract (English)
While Multi-Task Learning (MTL) is essential for adapting Large Language Models (LLMs) to diverse domains, prevailing LoRA-based methods rely on complex routing mechanisms that partition task-specific knowledge. In this work, we reveal that such routing-based designs are prone to a training-inference discrepancy, where stochastic routing decisions under distribution shifts compromise inference stability. Driven by a second-order Taylor analysis that exposes the instability induced by routing variance, we challenge the training-inference discrepancy and propose Consistency-Driven Low-Rank Adaptation (CD-LoRA). By eliminating routers entirely, CD-LoRA employs a consistency-driven alignment mechanism to enforce representation congruence across tasks in a shared low-rank space. This paradigm fosters robust, task-agnostic features without explicit partitioning overhead. Extensive experiments show that CD-LoRA consistently outperforms state-of-the-art multi-adapter baselines, offering a simpler, router-free, and more stable solution for multi-task PEFT. The code is available at the anonymous link https://github.com/zhaqian21/CD-LoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。