让每个令牌自行选择适配器，实现模型能力的模块化组合。
MoLoRA: Composable Specialization via Per-Token Adapter Routing
- 按令牌分配适配器，而非整段序列，支持多模态与混合任务。
- 小模型MoLoRA在4个推理任务上超越大模型，体积小4.7倍。
- 可独立训练、随时加载新功能，无需重新训练整个模型。
多适配器服务系统将整个序列路由到单一适配器，当请求跨越多个领域时会强制做出选择。这一假设在两种重要场景中失效：（1）多模态生成，同一序列中的文本和图像令牌需要不同适配器；（2）混合能力请求如“写代码解方程”，需多个专用适配器协同。我们提出按令牌路由机制，根据词汇结构（多模态模型）或学习到的门控机制（语义专业化）将单个令牌分配给适配器。该方法在理论上最优，处理N个令牌仅需工作量N，而传统每序列路由需K·N（K为适配器类型数）。核心贡献是MoLoRA（LoRA混合），实现可组合的专业化：加载多个领域专用适配器，由学习的路由器按令牌选择合适适配器。实验表明，专业化显著优于规模：MoLoRA使Qwen3-1.7B在四个推理基准上超过Qwen3-8B，体积仅为后者的1/4.7。这实现了推理时的模块化专家能力：可独立训练专注式LoRA，组合使用无需重训，新增能力只需加载新适配器。
原文摘要 · Abstract (English)
Multi-adapter serving systems route entire sequences to a single adapter, forcing a choice when requests span multiple domains. This assumption fails in two important settings: (1) multimodal generation, where text and image tokens require different adapters within the same sequence, and (2) mixed-capability requests like "write code to solve this equation," which need expertise from multiple specialized adapters. We introduce per-token routing, which routes individual tokens to adapters based on either vocabulary structure (for multimodal models) or learned gating (for semantic specialization). Per-token routing is provably optimal, achieving work N for N tokens versus K \cdot N for per-sequence routing with K adapter types. Our key contribution is MoLoRA (Mixture of LoRA), which enables composable specialization: load multiple domain-specific adapters and let a learned router select the appropriate adapter per-token. We demonstrate that specialization dramatically beats scale: MoLoRA enables Qwen3-1.7B to exceed Qwen3-8B across four reasoning benchmarks while being 4.7x smaller. This enables modular expertise at inference time: train focused LoRAs independently, combine them without retraining, and add new capabilities by simply loading new adapters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。