让视觉Transformer的令牌实现亚像素定位，提升效率与精度
SPoT: Subpixel Placement of Tokens in Vision Transformers
- 令牌位置从离散网格扩展到连续亚像素空间
- 理想定位下推理所需令牌数大幅减少，性能显著提升
- 适合追求高效、可解释视觉模型的研究者与应用
视觉Transformer天然支持稀疏性，但传统分块令牌化将特征限制在离散的图像块网格中，阻碍了对稀疏场景的充分利用，导致权衡困境。本文提出亚像素令牌定位（SPoT），一种新型令牌化策略，使令牌可在图像中连续定位，有效突破网格限制。通过设计的基于预言机的搜索方法，我们发现理想亚像素定位能带来显著性能提升，大幅降低推理时所需的令牌数量。SPoT为灵活、高效且可解释的ViT架构提供了新方向，将稀疏性重新定义为战略优势而非限制。
原文摘要 · Abstract (English)
Vision Transformers naturally accommodate sparsity, yet standard tokenization methods confine features to discrete patch grids. This constraint prevents models from fully exploiting sparse regimes, forcing awkward compromises. We propose Subpixel Placement of Tokens (SPoT), a novel tokenization strategy that positions tokens continuously within images, effectively sidestepping grid-based limitations. With our proposed oracle-guided search, we uncover substantial performance gains achievable with ideal subpixel token positioning, drastically reducing the number of tokens necessary for accurate predictions during inference. SPoT provides a new direction for flexible, efficient, and interpretable ViT architectures, redefining sparsity as a strategic advantage rather than an imposed limitation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。