调整图像块顺序可显著提升视觉模型性能
REOrdering Patches Improves Vision Models
- 提出两阶段框架REOrder，自动寻找最优图像块排列方式
- 在ImageNet-1K上准确率提升最高达3.01%，功能地图数据集提升13.35%
- 适用于对输入顺序敏感的长序列视觉模型，如改进Transformer
序列模型（如Transformer）需将输入表示为一维序列。在视觉任务中，通常采用固定的行优先（栅格扫描）顺序展开图像。尽管全自注意力机制具有置换等变性，但现代长序列Transformer越来越多依赖架构近似，破坏了这种不变性，导致对图像块顺序敏感。我们发现，在此类设置下，图像块顺序显著影响模型性能，简单的替代方案如列优先或希尔伯特曲线排列即可带来明显的准确率变化。为此，我们提出REOrder，一个两阶段框架，用于发现任务最优的图像块排列。首先，通过评估不同图像块序列的可压缩性，建立信息论先验；其次，使用REINFORCE优化普拉克特-卢斯策略，学习排列上的策略。该方法能高效地在组合排列空间中进行学习。REOrder在ImageNet-1K上相比行优先排序提升最高达3.01%的准确率，在功能地图世界（Functional Map of the World）上提升13.35%。
原文摘要 · Abstract (English)
Sequence models such as transformers require inputs to be represented as one-dimensional sequences. In vision, this typically involves flattening images using a fixed row-major (raster-scan) order. While full self-attention is permutation-equivariant, modern long-sequence transformers increasingly rely on architectural approximations that break this invariance and introduce sensitivity to patch ordering. We show that patch order significantly affects model performance in such settings, with simple alternatives like column-major or Hilbert curves yielding notable accuracy shifts. Motivated by this, we propose REOrder, a two-stage framework for discovering task-optimal patch orderings. First, we derive an information-theoretic prior by evaluating the compressibility of various patch sequences. Then, we learn a policy over permutations by optimizing a Plackett-Luce policy using REINFORCE. This approach enables efficient learning in a combinatorial permutation space. REOrder improves top-1 accuracy over row-major ordering on ImageNet-1K by up to 3.01% and Functional Map of the World by 13.35%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。