改进图像预训练：揭示噪声与掩码结合的关键条件
Improving Generative Pre-Training: An In-depth Study of Masked Image Modeling and Denoising Models
- 在编码器内特征空间引入噪声并分离无噪与掩码特征
- 显著提升细粒度识别等任务的预训练性能
- 适合研究视觉模型融合生成与掩码建模的学者
本文深入研究了添加噪声对深度网络预训练的影响。尽管受潜在去噪扩散模型启发，多种方法尝试将加性噪声引入预训练，但与掩码图像建模结合时，在识别任务上收益有限。为此，我们分析其原因并发现三个关键条件：噪声污染与恢复必须在编码器内进行，噪声需作用于特征空间，且需显式分离被噪声干扰与被掩码的令牌。基于此，我们构建的新方法在多种识别任务中实现性能提升，尤其在需要细粒度、高频信息的任务中表现更优。
原文摘要 · Abstract (English)
In this work, we dive deep into the impact of additive noise in pre-training deep networks. While various methods have attempted to use additive noise inspired by the success of latent denoising diffusion models, when used in combination with masked image modeling, their gains have been marginal when it comes to recognition tasks. We thus investigate why this would be the case, in an attempt to find effective ways to combine the two ideas. Specifically, we find three critical conditions: corruption and restoration must be applied within the encoder, noise must be introduced in the feature space, and an explicit disentanglement between noised and masked tokens is necessary. By implementing these findings, we demonstrate improved pre-training performance for a wide range of recognition tasks, including those that require fine-grained, high-frequency information to solve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。