用硬选择路由组合冻结的推理LoRA,高效保持专家能力
Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs

- 通过硬选路由在每个词元上仅激活一个推理LoRA专家
- 相比软混合方法,参数量减少但性能相当
- 适合需要轻量化多领域适配的场景
将独立训练的冻结LoRA适配器组合成大型语言模型,有助于多领域适应,尤其在原始数据不可共享时。传统方法采用类似MoE的路由机制,但对冻结适配器使用软加权组合会改变各LoRA模块原本的单位尺度增量更新。本文提出硬路由混合推理LoRA(Hard-Routed MoR-LoRA),采用两阶段框架:首先通过可验证反馈的强化学习训练出领域特定的推理专家;随后冻结所有专家,从中蒸馏推理轨迹,并仅训练一个轻量级共享路由器和一个小注意力LoRA进行集成。路由器采用硬top-1路由,每词元选择单一专家,借助直通估计器实现梯度训练。在五个基准测试、多个模型规模及多种模型架构上实验表明,该方法在保持专家行为的同时,所需可训练参数远少于软路由基线。分析进一步显示,归一化软混合常将大部分路由权重集中于单个专家,表明硬路由提供了对冻结LoRA专家组合的简洁高效抽象。
原文摘要 · Abstract (English)
Composing independently trained LoRA adapters into a single large language model is useful for multi-domain adaptation, especially when the original training data cannot be shared. A common approach is to use MoE-style routing over LoRA experts, but for frozen pretrained adapters, soft weighted combinations can change the unit-scale additive update under which each LoRA module was originally trained. We propose \textbf{Hard-Routed MoR-LoRA}, a two-stage framework for composing frozen reasoning LoRA experts through unit-scale hard selection. First, domain-specific LoRA adapters are trained independently using reinforcement learning from verifiable feedback to obtain reasoning experts. Then, all experts are frozen, reasoning traces are distilled from them, and only a lightweight shared router together with a small attention LoRA is trained for integration. The router selects exactly one expert per token using hard top-1 routing, while a straight-through estimator enables gradient-based training. Experiments across five benchmarks, multiple model scales, and additional model families show that Hard-Routed MoR-LoRA preserves expert behavior while requiring substantially fewer trainable parameters than soft-routing mixture baselines. Our analysis further shows that normalized soft mixtures often concentrate most routing mass on a single expert, suggesting that hard unit-scale routing provides a simple and efficient abstraction for frozen LoRA expert composition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。