解决视频扩散模型中LoRA迁移的风格退化问题
Exploring Data-Free LoRA Transferability for Video Diffusion Models
- 通过分析权重空间发现谱干扰是迁移失败根源
- 提出CASA框架，动态调节谱密度以恢复对齐
- 无需数据即可提升迁移效果，适合模型复用场景
基于步骤蒸馏或因果蒸馏的视频扩散模型已取得显著性能。然而，将现有LoRA适配到这些变体仍面临重大挑战，主要源于权重空间不匹配。我们观察到直接应用会导致风格退化和结构崩溃，但其内在机制尚不清楚。为填补这一空白，我们深入分析权重空间，发现不兼容性源于在奇异子空间上定义的共享功能簇中的谱干扰。具体而言，尽管两种范式均遵守谱刚性，但它们建立的路由路径存在冲突，导致建设性过载或破坏性抵消。为此，我们提出无数据框架Cluster-Aware Spectral Arbitration (CASA)，根据谱密度动态协调保护目标流形与恢复LoRA对齐之间的关系。大量实验表明，CASA能有效缓解伪影并恢复LoRA功能。代码已开源：https://github.com/Noahwangyuchen/CASA
原文摘要 · Abstract (English)
Video diffusion models leveraging step distillation or causal distillation have achieved remarkable performance. However, adapting existing LoRAs to these variants remains a critical challenge due to weight space mismatches. We observe that direct application leads to style degradation and structural collapse, yet the underlying mechanisms remain poorly understood. To fill this gap, we delve into the weight space and identify that the incompatibility stems from spectral interference within shared functional clusters defined over singular subspaces. Specifically, our analysis reveals that while both paradigms respect spectral rigidity, they establish conflicting routing pathways that clash through constructive overload or destructive cancellation. To address this issue, we propose Cluster-Aware Spectral Arbitration (CASA), a data-free framework that dynamically arbitrates between safeguarding the target's manifold and restoring LoRA alignment based on spectral density. Extensive experiments demonstrate that CASA effectively mitigates artifacts and revives LoRA functionality. Our code is available at https://github.com/Noahwangyuchen/CASA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。