arXiv:2511.17979cs.CV2025-11被引 6

通过频率能量机制优化扩散模型微调,提升适应性与稳定性。

FeRA: Frequency-Energy Constrained Routing for Effective Diffusion Adaptation Fine-Tuning

  • 基于隐空间频率能量分布设计动态路由机制
  • 三组件协同实现跨频段稳定微调,精度提升显著
  • 兼容多种模型架构,推理时可动态调整路由

扩散模型在生成建模中取得显著成功,但如何有效将大型预训练模型适配到新任务仍具挑战。本文重新审视去噪过程中扩散模型的重构行为,揭示了控制该过程的内在频率能量机制。基于此,提出FeRA——一种以频率驱动的微调框架,使参数更新与扩散模型固有的频率能量演变对齐。FeRA构建了一个完整的频率能量框架,包含三个协同组件:(i) 紧凑的频率能量指示器,刻画潜在表示的带宽能量分布;(ii) 软频率路由器,自适应融合多个频率特异性适配专家;(iii) 频率能量一致性正则化,稳定扩散优化并保证各频段间适配的一致性。路由机制在训练和推理阶段均生效,推理时根据潜在频率能量动态决定。该方法可无缝集成于基于适配器的微调方案,且在不同扩散主干网络和分辨率下具有良好泛化能力。通过与频率能量机制对齐,FeRA提供了一种简单、稳定且兼容的扩散模型有效适应范式。

原文摘要 · Abstract (English)

Diffusion models have achieved remarkable success in generative modeling, yet how to effectively adapt large pretrained models to new tasks remains challenging. We revisit the reconstruction behavior of diffusion models during denoising to unveil the underlying frequency energy mechanism governing this process. Building upon this observation, we propose FeRA, a frequency driven fine tuning framework that aligns parameter updates with the intrinsic frequency energy progression of diffusion. FeRA establishes a comprehensive frequency energy framework for effective diffusion adaptation fine tuning, comprising three synergistic components: (i) a compact frequency energy indicator that characterizes the latent bandwise energy distribution, (ii) a soft frequency router that adaptively fuses multiple frequency specific adapter experts, and (iii) a frequency energy consistency regularization that stabilizes diffusion optimization and ensures coherent adaptation across bands. Routing operates in both training and inference, with inference time routing dynamically determined by the latent frequency energy. It integrates seamlessly with adapter based tuning schemes and generalizes well across diffusion backbones and resolutions. By aligning adaptation with the frequency energy mechanism, FeRA provides a simple, stable, and compatible paradigm for effective and robust diffusion model adaptation.

扩散模型微调频率分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。