让图像输入自适应聚焦，减少计算量并提升精度。
SpiralFovea: Input-Adaptive Foveated Tokenization as a Third Lever of Resource-Adaptive Inference
- 根据图像内容动态生成多尺度螺旋令牌，替代固定网格
- 精度提升1.7-2.1个百分点，输入令牌减少60%，注意力计算量降84%
- 适合自监督大模型，尤其在需要全局位置先验的场景
主流自适应推理技术多调整模型行为，如提前退出、MoE路由、键值缓存压缩或动态注意力稀疏化，但输入到主干网络前的分块方式仍为固定网格，与图像内容无关。本文提出SpiralFovea，一种无参数、输入自适应的分块方法：令牌身份、位置、尺度和数量均由局部视觉熵决定，且在调用任何主干参数前完成选择。在内容热点区域，多尺度螺旋环生成最多78个补丁，取代标准ViT的196个补丁。在四个细粒度基准测试中，SpiralFovea实现1.7-2.1个百分点的精度提升，输入令牌减少60%，每层Transformer的自注意力浮点运算量下降84%，吞吐量提高18-29%。在CUB-200-2011 Genus上的控制消融实验显示，性能增益与主干模型全局位置先验强度呈负相关，表明自监督基础模型是输入自适应分块最受益的场景。
原文摘要 · Abstract (English)
Most adaptive-inference techniques for foundation models change what the model does - early exit, MoE routing, KV-cache compression, dynamic attention sparsity. The input that hits the backbone, however, remains a fixed-grid tokenisation indifferent to image content. We argue that this is a missed lever. We present SpiralFovea, a parameter-free, input-adaptive tokeniser in which token identity, location, scale, and count are all functions of local visual entropy and selection completes before any backbone parameter is queried. Around content-driven hotspot anchors, multi-scale spiral rings produce <= 78 patches that replace the standard 196-patch ViT grid at the input stage. Across four canonical fine-grained benchmarks, SpiralFovea yields +1.7-2.1 pp accuracy with a 60% reduction in input tokens, an 84% reduction in self-attention FLOPs at every transformer layer, and 18-29% throughput gains over the matched static tokenisation baseline. A controlled ablation on CUB-200-2011 Genus across four backbones reveals a clean diagnostic: the gain magnitude tracks inversely with the strength of the backbone's whole-image positional prior, isolating self-supervised foundation models as the regime where input-adaptive tokenisation is most valuable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。