用混合低秩适配器提升递归变压器的表达能力。
Improving Recursive Transformers with Mixture of LoRAs
- 在共享前馈网络中插入低秩适配专家，实现条件化权重调制。
- 5000万至1.2亿参数模型在多项任务上超越更大全参数模型。
- 适配器合并技术让部署更高效，适合资源受限场景。
递归变压器通过参数共享减少模型规模，但削弱了各层表达能力。我们提出混合低秩适配器（MoL），一种轻量级条件计算机制，在共享前馈网络中嵌入低秩适配专家（LoRA），实现不解除主干参数前提下的令牌条件权重调制。相比以往固定或外接适配器的方法，该机制更具灵活性。我们预训练了一个现代化递归架构ModernALBERT，整合旋转位置编码、GeGLU激活函数、FlashAttention及基于蒸馏的初始化方式。在GLUE、SQuAD-v2和BEIR等基准上，5000万至1.2亿参数的ModernALBERT表现优于同类紧凑模型，并超越更大的全参数基线模型。我们还提出一种专家合并方法，可在推理阶段将MoL压缩为单个适配器，同时保持精度，便于高效部署。结果表明，条件权重调制能有效恢复递归变压器在激进参数共享下的表达力损失。
原文摘要 · Abstract (English)
Parameter sharing in recursive transformers reduces model size but collapses layer-wise expressivity. We propose Mixture of LoRAs (MoL), a lightweight conditional-computation mechanism that inserts Low-Rank Adaptation (LoRA) experts inside a shared feed-forward network (FFN). MoL enables token-conditional weight-space modulation of the shared FFN without untying backbone parameters, unlike prior approaches that add fixed or externally attached adapters. We pretrain a modernised recursive architecture, ModernALBERT, integrating rotary embeddings, GeGLU, FlashAttention, and a distillation-based initialisation. Across GLUE, SQuAD-v2, and BEIR, ModernALBERT (50M--120M) achieves state-of-the-art performance among compact models and surpasses larger fully parameterised baselines. We also propose an expert-merging procedure that compresses MoL into a single adapter at inference while preserving accuracy, enabling efficient deployment. Our results show that conditional weight-space modulation effectively restores the expressivity lost under aggressive parameter sharing in recursive transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。