通过分析图像块的依赖关系，找到高效建模的最优生成顺序。
Collapse of Patches: Ranking Image Patches for Efficient Visual Modeling
- 用自编码器学习关键图像块，按重要性排序生成顺序。
- 仅用22%的高阶块即可达到高分类准确率。
- 适用于生成和分类，提升视觉模型效率。
观察图像中的某些区域会降低其他区域的不确定性，其特征分布熵随之下降，类似于量子力学中波函数坍缩。我们称此现象为图像块坍缩。为识别目标区域坍缩时最依赖的块，我们训练一个自编码器，软性选择重建时最具信息量的块集合。通过计算每个块的PageRank得分，构建出最优的图像生成顺序。实验表明，遵循该顺序可提升多种掩码图像建模方法：在自回归图像生成中微调生成序列可显著提升性能；此外，我们提出一种新图像分类范式，仅向视觉变压器（Vision Transformers）暴露坍缩顺序中排名靠前的块。结果显示，仅需22%的此类关键块，即可实现高精度分类。这些发现揭示了图像块坍缩作为新型视觉建模视角的潜力，有助于提升视觉任务的效率。
原文摘要 · Abstract (English)
Observing certain patches in an image reduces the uncertainty of others. Their realization lowers the distribution entropy of each remaining patch feature, analogous to collapsing a particle's wave function in quantum mechanics. This phenomenon can intuitively be called patch collapse. To identify which patches are most relied on during a target region's collapse, we learn an autoencoder that softly selects a subset of informative patches during reconstruction. Graphing these learned dependencies for each patch's PageRank score reveals the optimal patch order to realize an image. We show that respecting this order benefits various masked image modeling methods. First, autoregressive image generation can be boosted by finetuning with the ordered generation sequence. Second, we introduce a new setup for image classification by exposing Vision Transformers only to high-rank patches in the collapse order. Seeing 22% of such patches is sufficient to achieve high accuracy. With these experiments, we propose patch collapse as a novel image modeling perspective that promotes vision efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。