用遮蔽图像训练模型,高效学习概念并支持可控生成。
MCM: Multi-layer Concept Map for Efficient Concept Learning from Masked Images
- 通过多层关联编码器与解码器,利用反向梯度更新概念标记。
- 仅用75%图像块训练,计算成本降低且概念预测性能提升。
- 可编辑潜在空间概念标记,实现基于遮蔽图的定向图像生成。
在视觉任务中,如概念学习,传统方法通常依赖完整图像,而掩码策略在自然语言处理中已广泛应用。本文提出首个基于掩码图像的高效概念学习方法——多层概念图(MCM)。该方法采用非对称架构,建立编码器与解码器不同层间的关联,通过重建任务的反向梯度更新概念标记。不同粒度的概念标记既可填补遮蔽图像块,也可引导重建结果体现特定概念。实验表明,MCM在少于75%图像块上训练,显著降低计算开销,同时提升概念预测性能。此外,通过修改潜在空间中的概念标记,可在遮蔽图像基础上生成符合指定概念的图像;调节测试时的掩码比例,可按比例融合可见区域与概念信息,实现多样化的重建结果。
原文摘要 · Abstract (English)
Masking strategies commonly employed in natural language processing are still underexplored in vision tasks such as concept learning, where conventional methods typically rely on full images. However, using masked images diversifies perceptual inputs, potentially offering significant advantages in concept learning with large-scale Transformer models. To this end, we propose Multi-layer Concept Map (MCM), the first work to devise an efficient concept learning method based on masked images. In particular, we introduce an asymmetric concept learning architecture by establishing correlations between different encoder and decoder layers, updating concept tokens using backward gradients from reconstruction tasks. The learned concept tokens at various levels of granularity help either reconstruct the masked image patches by filling in gaps or guide the reconstruction results in a direction that reflects specific concepts. Moreover, we present both quantitative and qualitative results across a wide range of metrics, demonstrating that MCM significantly reduces computational costs by training on fewer than 75% of the total image patches while enhancing concept prediction performance. Additionally, editing specific concept tokens in the latent space enables targeted image generation from masked images, aligning both the visible contextual patches and the provided concepts. By further adjusting the testing time mask ratio, we could produce a range of reconstructions that blend the visible patches with the provided concepts, proportional to the chosen ratios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。