arXiv:2603.09955cs.CVcs.LG2026-03

通过分层掩码重建,让模型从语义到像素逐步理解图像。

From Semantics to Pixels: Coarse-to-Fine Masked Autoencoders for Hierarchical Visual Understanding

  • 分三阶段重建:场景语义→物体实例→像素细节,逐级增强理解。
  • 在ImageNet-1K上分类准确率提升至87.2%,检测与分割性能显著超越基线。
  • 适合追求视觉表征鲁棒性的研究者,尤其关注多粒度建模的场景。

自监督视觉预训练方法面临内在矛盾:对比学习(CL)捕捉全局语义但丢失细粒度细节,而掩码图像建模(MIM)保留局部纹理却因语义无关的随机掩码导致注意力漂移。我们提出C2FMAE,一种从粗到细的掩码自编码器,通过显式学习三个数据粒度的层次化视觉表示——语义掩码(场景级)、实例掩码(物体级)和RGB图像(像素级)。两个协同创新强化自顶向下的学习原则:首先,级联解码器依次重建从场景语义到物体实例再到像素细节,建立解码器无法捕捉的跨粒度依赖关系;其次,渐进式掩码课程动态转移训练重点,从语义引导到实例引导,最终到随机掩码,构建从全局上下文到局部特征的结构化学习路径。为支持该框架,我们构建了一个大规模多粒度数据集,包含128万张ImageNet-1K图像的高质量伪标签。大量实验表明,C2FMAE在图像分类、目标检测和语义分割任务上均取得显著提升,验证了其层次化设计在学习更鲁棒、泛化性更强表征方面的有效性。

原文摘要 · Abstract (English)

Self-supervised visual pre-training methods face an inherent tension: contrastive learning (CL) captures global semantics but loses fine-grained detail, while masked image modeling (MIM) preserves local textures but suffers from "attention drift" due to semantically-agnostic random masking. We propose C2FMAE, a coarse-to-fine masked autoencoder that resolves this tension by explicitly learning hierarchical visual representations across three data granularities: semantic masks (scene-level), instance masks (object-level), and RGB images (pixel-level). Two synergistic innovations enforce a strict top-down learning principle. First, a cascaded decoder sequentially reconstructs from scene semantics to object instances to pixel details, establishing explicit cross-granularity dependencies that parallel decoders cannot capture. Second, a progressive masking curriculum dynamically shifts the training focus from semantic-guided to instance-guided and finally to random masking, creating a structured learning path from global context to local features. To support this framework, we construct a large-scale multi-granular dataset with high-quality pseudo-labels for all 1.28M ImageNet-1K images. Extensive experiments show that C2FMAE achieves significant performance gains on image classification, object detection, and semantic segmentation, validating the effectiveness of our hierarchical design in learning more robust and generalizable representations.

视觉理解自监督掩码建模层次表示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。