用频率调控扩散过程,防止测试时自适应中的灾难性遗忘。
FOCUS: Frequency-Optimized Conditioning of DiffUSion Models for mitigating catastrophic forgetting during Test-Time Adaptation
- 基于频域先验条件化扩散去噪过程,保留语义信息。
- 在15种噪声下跨3个数据集达最优平均性能。
- 轻量网络+伪标签,适合资源受限的持续学习场景。
测试时自适应使模型能适应动态变化的数据分布,但如何在适应新域的同时保留旧任务知识仍具挑战,因适应可能引发知识遗忘。为此,我们提出FOCUS,一种基于频率的条件化方法,嵌入扩散驱动的输入自适应框架。通过学习的空间自适应频率先验,在扩散去噪的逆向步骤中对齐高频细节与低频语义,有效保护密集预测任务的关键语义信息。该方法采用轻量级Y形频率预测网络(Y-FPN),可分离噪声图像中的高低频成分,显著降低计算开销。训练时使用新颖的数据增强方法FrequencyMix,通过扰动图像在多频带上的表现,提升模型对多样化损坏的鲁棒性。我们在15种不同类型的噪声及三个数据集上验证了FOCUS在语义分割与单目深度估计上的有效性,达到当前最优平均性能。此外,由FOCUS去噪图像生成的伪标签可为现有自适应方法提供额外监督,即使在稀疏、间歇性标注条件下,仍能有效缓解最新自适应方法的灾难性遗忘问题。
原文摘要 · Abstract (English)
Test-time adaptation enables models to adapt to evolving domains. However, balancing the tradeoff between preserving knowledge and adapting to domain shifts remains challenging for model adaptation methods, since adapting to domain shifts can induce forgetting of task-relevant knowledge. To address this problem, we propose FOCUS, a novel frequency-based conditioning approach within a diffusion-driven input-adaptation framework. Utilising learned, spatially adaptive frequency priors, our approach conditions the reverse steps during diffusion-driven denoising to preserve task-relevant semantic information for dense prediction. FOCUS leverages a trained, lightweight, Y-shaped Frequency Prediction Network (Y-FPN) that disentangles high and low frequency information from noisy images. This minimizes the computational costs involved in implementing our approach in a diffusion-driven framework. We train Y-FPN with FrequencyMix, a novel data augmentation method that perturbs the images across diverse frequency bands, which improves the robustness of our approach to diverse corruptions. We demonstrate the effectiveness of FOCUS for semantic segmentation and monocular depth estimation across 15 corruption types and three datasets, achieving state-of-the-art averaged performance. In addition to improving standalone performance, FOCUS complements existing model adaptation methods since we can derive pseudo labels from FOCUS-denoised images for additional supervision. Even under limited, intermittent supervision with the pseudo labels derived from the FOCUS denoised images, we show that FOCUS mitigates catastrophic forgetting for recent model adaptation methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。