用神经调制思想让LoRA动态适应上下文，提升多任务学习效果。
NeuroLoRA: Context-Aware Neuromodulation for Parameter-Efficient Multi-Task Adaptation
- 基于生物神经调制机制设计可学习的门控模块，动态调节专家选择。
- 在多个数据集上超越基线，连续学习时性能下降减少42%。
- 适合需要高效多任务适配的大模型场景，如智能助手、知识系统。
参数高效微调（PEFT）技术，特别是低秩适配（LoRA），已成为将大语言模型（LLMs）适配到下游任务的关键方法。尽管近期的FlyLoRA框架通过受生物启发的稀疏随机投影有效缓解了参数干扰，但其依赖静态的基于幅度的路由机制，对输入上下文不敏感。本文提出NeuroLoRA，一种受生物神经调制启发的基于专家混合（MoE）的LoRA框架——通过上下文动态调节神经元兴奋性来实现专家选择。NeuroLoRA在保持冻结随机投影计算效率的同时，引入轻量级可学习的神经调制门控，在专家选择前上下文地重缩放投影空间。我们进一步提出对比正交性损失，显式强化专家子空间间的分离，从而提升任务解耦与持续学习能力。在MMLU、GSM8K和ScienceQA上的大量实验表明，NeuroLoRA在单任务微调、多任务模型合并及顺序持续学习场景中均持续优于FlyLoRA及其他强基线，同时保持相近的参数效率。
原文摘要 · Abstract (English)
Parameter-Efficient Fine-Tuning (PEFT) techniques, particularly Low-Rank Adaptation (LoRA), have become essential for adapting Large Language Models (LLMs) to downstream tasks. While the recent FlyLoRA framework successfully leverages bio-inspired sparse random projections to mitigate parameter interference, it relies on a static, magnitude-based routing mechanism that is agnostic to input context. In this paper, we propose NeuroLoRA, a novel Mixture-of-Experts (MoE) based LoRA framework inspired by biological neuromodulation -- the dynamic regulation of neuronal excitability based on context. NeuroLoRA retains the computational efficiency of frozen random projections while introducing a lightweight, learnable neuromodulation gate that contextually rescales the projection space prior to expert selection. We further propose a Contrastive Orthogonality Loss to explicitly enforce separation between expert subspaces, enhancing both task decoupling and continual learning capacity. Extensive experiments on MMLU, GSM8K, and ScienceQA demonstrate that NeuroLoRA consistently outperforms FlyLoRA and other strong baselines across single-task adaptation, multi-task model merging, and sequential continual learning scenarios, while maintaining comparable parameter efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。