让参数高效微调实现按令牌分工,不增新参数却更省资源。
Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning
- 利用现有适配器作隐式专家,用k-means聚类路由令牌
- 14文本/14图像/19视频任务上性能媲美MoE方法,参数少7-29倍
- 无需训练参数,可适配任意基于适配器的微调方法
参数高效微调的混合专家变体能实现按令牌专业化,但引入了额外的可训练路由器和专家参数,增加了内存占用和训练成本,违背了参数高效的核心目标。本文提出Monkey Jump,将混合专家式专业化引入参数高效微调,无需新增专家或路由器的可训练参数。该方法不添加新适配器,而是将Transformer块中已有的适配器(如查询、键、值、上采样、下采样投影)视为隐式专家,并通过使用指数移动平均聚类中心的k-means聚类实现令牌路由,无需梯度与学习参数。理论上,令牌级路由提升了模型表达能力,可避免共享适配器中的抵消效应。在涵盖14个文本、14个图像和19个视频基准的多任务实验中,Monkey Jump在性能上可媲美基于混合专家的方法,同时减少7至29倍的可训练参数,内存消耗降低最高达48%,训练速度提升1.5至2倍。该方法具有架构无关性,可应用于任何基于适配器的参数高效微调方法。
原文摘要 · Abstract (English)
Mixture-of-experts variants of parameter-efficient fine-tuning enable per-token specialization, but they introduce additional trainable routers and expert parameters, increasing memory usage and training cost. This undermines the core goal of parameter-efficient fine-tuning. We propose Monkey Jump, a method that brings mixture-of-experts-style specialization to parameter-efficient fine-tuning without introducing extra trainable parameters for experts or routers. Instead of adding new adapters as experts, Monkey Jump treats the adapters already present in each Transformer block (such as query, key, value, up, and down projections) as implicit experts and routes tokens among them. Routing is performed using k-means clustering with exponentially moving averaged cluster centers, requiring no gradients and no learned parameters. We theoretically show that token-wise routing increases expressivity and can outperform shared adapters by avoiding cancellation effects. Across multi-task experiments covering 14 text, 14 image, and 19 video benchmarks, Monkey Jump achieves competitive performance with mixture-of-experts-based parameter-efficient fine-tuning methods while using 7 to 29 times fewer trainable parameters, up to 48 percent lower memory consumption, and 1.5 to 2 times faster training. Monkey Jump is architecture-agnostic and can be applied to any adapter-based parameter-efficient fine-tuning method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。