融合像素与隐变量建模，提升视觉表征能力
PiLaMIM: Toward Richer Visual Representations by Integrating Pixel and Latent Masked Image Modeling
- 单编码器双解码器架构，同时重建像素和隐变量
- 在ImageNet上分类准确率超MAE 1.2%，优于I-JEPA等基线
- 适合需要兼顾细节与语义的视觉任务，如图像识别
掩码图像建模（MIM）主要分为像素级建模（Pixel MIM）和隐变量级建模（Latent MIM），分别侧重低层视觉细节（如颜色、纹理）和高层语义。然而两者特性差异导致在依赖特定层次特征的任务中表现受限。为此，我们提出PiLaMIM，一种统一框架，结合两种方法优势。该方法采用单一编码器与两个独立解码器：一个重建像素值，另一个重建隐变量表示，实现高低层特征同步捕获。此外，通过将CLS token融入重建过程，增强全局上下文信息聚合，提升语义表达能力。大量实验表明，PiLaMIM在多数情况下优于MAE、I-JEPA和BootMAE等主流基线模型，验证了其在提取更丰富视觉表示方面的有效性。
原文摘要 · Abstract (English)
In Masked Image Modeling (MIM), two primary methods exist: Pixel MIM and Latent MIM, each utilizing different reconstruction targets, raw pixels and latent representations, respectively. Pixel MIM tends to capture low-level visual details such as color and texture, while Latent MIM focuses on high-level semantics of an object. However, these distinct strengths of each method can lead to suboptimal performance in tasks that rely on a particular level of visual features. To address this limitation, we propose PiLaMIM, a unified framework that combines Pixel MIM and Latent MIM to integrate their complementary strengths. Our method uses a single encoder along with two distinct decoders: one for predicting pixel values and another for latent representations, ensuring the capture of both high-level and low-level visual features. We further integrate the CLS token into the reconstruction process to aggregate global context, enabling the model to capture more semantic information. Extensive experiments demonstrate that PiLaMIM outperforms key baselines such as MAE, I-JEPA and BootMAE in most cases, proving its effectiveness in extracting richer visual representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。