让多个低秩专家在路由前互相交流，提升大模型微调的稳定性与效率。
TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models
- 引入轻量级对话模块，让低秩专家在路由前交换信息，打破独立假设。
- 在多任务上优于传统LoRA和MoELoRA，路由更均衡，参数效率更高。
- 适合追求高效微调且关注专家协同的LLM研究者与工程师。
低秩适配（LoRA）实现了大语言模型的参数高效微调，近期的专家混合（MoE）扩展通过动态组合多个LoRA专家进一步提升了灵活性。然而，现有MoE增强的LoRA方法假设专家相互独立，常导致路由不稳定和专家主导问题。本文提出 extbf{TalkLoRA}，一种通信感知的MoELoRA框架，通过在路由前引入专家级通信来放松独立性假设。TalkLoRA为低秩专家配备轻量级对话模块，实现专家子空间间的可控信息交换，生成更稳健的全局路由信号。理论上，专家通信通过缓解扰动放大效应，平滑路由动态，同时严格推广现有MoELoRA架构。实验表明，TalkLoRA在多样化的语言理解与生成任务中持续优于基线，以相近参数预算实现更高参数效率与更均衡的专家路由。结果表明，结构化专家通信是基于MoE的参数高效适配的原理性有效改进。代码已开源：https://github.com/why0129/TalkLoRA。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of Large Language Models (LLMs), and recent Mixture-of-Experts (MoE) extensions further enhance flexibility by dynamically combining multiple LoRA experts. However, existing MoE-augmented LoRA methods assume that experts operate independently, often leading to unstable routing, expert dominance. In this paper, we propose \textbf{TalkLoRA}, a communication-aware MoELoRA framework that relaxes this independence assumption by introducing expert-level communication prior to routing. TalkLoRA equips low-rank experts with a lightweight Talking Module that enables controlled information exchange across expert subspaces, producing a more robust global signal for routing. Theoretically, we show that expert communication smooths routing dynamics by mitigating perturbation amplification while strictly generalizing existing MoELoRA architectures. Empirically, TalkLoRA consistently outperforms vanilla LoRA and MoELoRA across diverse language understanding and generation tasks, achieving higher parameter efficiency and more balanced expert routing under comparable parameter budgets. These results highlight structured expert communication as a principled and effective enhancement for MoE-based parameter-efficient adaptation. Code is available at https://github.com/why0129/TalkLoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。