arXiv:2607.00162cs.LG2026-07被引 2

让模型自己学该在什么域做参数更新,提升效率与多任务表现。

FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts

论文配图:FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts
图 1 · 摘自论文原文
  • 用可学习的分数阶傅里叶顺序,连接空间与频域,动态选择最优更新域。
  • 在多个大模型基准上优于现有方法,且激活参数量保持极低。
  • 专家间天然解耦,适合多任务微调,且学习到的域偏移有解释性。

参数高效微调(PEFT)在固定基底下重参数化权重更新:低秩适配器在空间域操作,而近期谱方法则在固定傅里叶域进行。我们提出,域的选择本身应作为可学习的设计自由度,且单一基底无法在所有任务、层或标记上保持最优。为此引入分数阶傅里叶混合专家(Fractional-Fourier Mixture of Experts),其每个专家携带一个可学习的分数阶傅里叶阶数,连续介于空间域(恢复标准LoRA)与傅里叶域(恢复谱适配器)之间。通过将令牌路由至不同位置的专家,模型能为每个低秩更新选择最紧凑的域;由于不同阶数的分数阶傅里叶算子互不相干,专家自然解耦,减少干扰并提升多任务组合能力。每专家仅一个标量阶数,使用独立优化器训练,变换采用复杂度为 $/mathcal{O}(d\ ext{log} d)$ 的啁啾-FFT近似,因此计算开销仅略高于标准MoE-LoRA。在LLaMA-3.1-8B与Qwen2.5-7B上的常识、数学、代码与知识基准测试中,该方法超越强基线(如FlyLoRA、FourierMoE、HMoRA),同时保持小规模活跃参数量;分析显示,学习到的阶数在任务与层间具有可解释的专门化特性。

原文摘要 · Abstract (English)

Parameter-efficient fine-tuning (PEFT) reparameterizes weight updates in a fixed basis: low-rank adapters operate in the spatial domain, while a recent line of spectral methods operates in a fixed Fourier domain. We argue that the choice of domain is itself a design degree of freedom that should be learned, and that no single basis is optimal across tasks, layers, or tokens. We introduce Fractional-Fourier Mixture of Experts, a mixture-of-experts adapter in which every expert carries a learnable fractional-Fourier order that continuously interpolates between the spatial domain (recovering vanilla LoRA) and the Fourier domain (recovering a spectral adapter). Routing tokens through experts that occupy different points on this spatial-spectral continuum lets the model place each low-rank update in the domain where it is most compact, and -- because fractional-Fourier operators of different orders are mutually incoherent -- makes the experts naturally decorrelated, which reduces interference and improves multi-task composition. The order is a single scalar per expert, trained with a separate optimizer, and the transform is computed with an $\mathcal{O}(d\log d)$ chirp--FFT surrogate, so Fractional-Fourier Mixture of Experts adds negligible cost over standard MoE-LoRA. Across commonsense, mathematical, code, and knowledge benchmarks on LLaMA-3.1-8B and Qwen2.5-7B, Fractional-Fourier Mixture of Experts improves over strong MoE-LoRA and spectral baselines -- including FlyLoRA, FourierMoE, and HMoRA -- while keeping the active-parameter budget small, and analysis shows that the learned orders specialize by task and layer in interpretable ways.

参数高效混合专家傅里叶微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。