解决扩散模型微调时风格混淆问题,让专家分工更早开始。
Ortho-Hydra: Orthogonalized Experts for DiT LoRA
- 用正交共享基+专家独占输出空间,打破初始对称性
- 训练初期即打破均匀路由,专家分化提前数百步发生
- 适合需要多风格微调的扩散模型用户
在多风格数据上微调扩散变换器(DiT)时,LoRA易出现风格混淆:单个低秩残差无法区分不同艺术家风格,优化器收敛到平均值。传统HydraLoRA虽引入多专家结构,但所有专家零初始化时梯度相同,路由器维持均匀先验,专家对称演化,实际等同于单个秩为$ r $的LoRA,代价却增至$ E{ imes} $。本文提出Ortho-Hydra,采用OFT风格的Cayley正交共享基,并从预训练权重的前$ Er $个左奇异向量中切分出每个专家的独占输出子空间。该设计确保初始阶段路由器各专家得分非退化,使专家分化在训练早期就获得梯度信号。通过对比两个基准(零初始化共享基与$σ=0.1$高斯抖动),在相同优化器、数据集和训练步数下,两者前1000步均未脱离均匀先验;而Ortho-Hydra在前几百步即开始去均匀化。报告聚焦于架构构建、冷启动机制及路由动态变化,不涉及多风格生成质量。代码见:https://github.com/sorryhyun/anima_lora。
原文摘要 · Abstract (English)
LoRA fine-tuning of diffusion transformers (DiT) on multi-style data suffers from \emph{style bleed}: a single low-rank residual cannot represent several distinct artist fingerprints, and the optimizer converges to their average. Mixture-of-experts LoRA in the HydraLoRA style replaces the up-projection with $E$ heads under a router, but when every expert is zero-initialized the router receives identical gradient from each head and remains at the uniform prior. The experts then evolve permutation-symmetrically, and the network trains as a single rank-$r$ LoRA at $E{\times}$ the cost. We present \textbf{Ortho-Hydra}, a re-parameterisation that combines an OFT-style Cayley-orthogonal shared basis with per-expert \emph{disjoint output subspaces} carved from the top-$(Er)$ left singular vectors of the pretrained weight. Disjointness makes the router's per-expert score non-degenerate at step~$0$, so specialization receives gradient signal before any expert has trained. We test the predicted deadlock on a DiT pipeline by comparing two HydraLoRA baselines, a zero-initialized shared-basis variant and the original $σ{=}0.1$ Gaussian-jitter mitigation, against Ortho-Hydra under a matched optimiser, dataset, and step budget. Neither baseline leaves the uniform prior within the first $1\text{k}$ steps; Ortho-Hydra begins de-uniformising within the first few hundred. End-task generation quality on multi-style data is out of scope; we report the construction, the cold-start mechanism, and the routing dynamics it changes. Code: https://github.com/sorryhyun/anima_lora.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。