用专家提示调优构建动态路径，让模型更灵活适应数据变化。
MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper
- 引入专家提示混合架构，动态选择适配数据的神经路径。
- 在SuperGLUE上平均准确率提升1.94%，激活提示减少79.25%。
- 适合需要高效适配多变数据分布的任务场景。
将深度神经网络视为流形映射器，预训练-微调范式可被理解为两阶段过程：预训练建立广泛知识基础，微调则调整模型参数以激活特定神经通路，使其对齐目标流形。尽管已有微调方法表现成功，但其固定的参数空间限制了动态激活合适通路的能力，难以灵活应对多样且动态变化的数据分布。为此，我们提出一种新方法——专家提示调优混合（MEPT），作为有效的高效流形映射框架。MEPT通过融合多个提示专家的专家混合架构，自适应学习多样化和非平稳的数据分布。实验表明，MEPT在SuperGLUE上优于多个先进参数高效基线，平均准确率提升1.94%，同时激活提示数量减少79.25%。其有效性得到流形学习理论分析支持，并通过神经激活路径可视化验证。代码已公开于https://runjia.tech/emnlp_mept/。
原文摘要 · Abstract (English)
Considering deep neural networks as manifold mappers, the pretrain-then-fine-tune paradigm can be interpreted as a two-stage process: pretrain establishes a broad knowledge base, and fine-tune adjusts the model parameters to activate specific neural pathways to align with the target manifold. Although prior fine-tuning approaches demonstrate success, their rigid parameter space limits their ability to dynamically activate appropriate neural pathways, rendering them ill-equipped to adapt flexibly to the diverse and evolving data distributions. In light of this view, we propose a novel approach, Mixture of Expert Prompt Tuning (MEPT), as an effective and efficient manifold-mapping framework. MEPT leverages the Mixture of Experts architecture by integrating multiple prompt experts to adaptively learn diverse and non-stationary data distributions. Empirical evaluations demonstrate that MEPT outperforms several state-of-the-art parameter efficient baselines on SuperGLUE, achieving notable improvements in mean accuracy (e.g., 1.94%) while significantly reducing activated prompts by 79.25%. The effectiveness of MEPT is further supported by theoretical insights from manifold learning and validated through neural activation pathway visualization results. Our code is avaliable at https://runjia.tech/emnlp_mept/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。