模仿人眼扫视,只处理图像关键区域,大幅节省内存和计算量。
A Sensorimotor Vision Transformer
- 基于人眼扫视机制,动态选择高信息量图像块进行处理。
- 在ImageNet-1k上以少量图像块实现与传统模型相当的准确率。
- 适合资源受限场景,为高效视觉模型提供生物启发新思路。
本文提出传感器运动视觉变换器(SMT),一种受人类扫视眼动启发的视觉模型,通过优先处理视觉输入中高显著性区域,提升计算效率并降低内存消耗。不同于传统模型对所有图像块均匀处理,SMT基于内在二维(i2D)特征(如角点、遮挡)识别并选择最具显著性的图像块,这些特征蕴含高信息量且符合人类注视模式。SMT架构利用这一生物原理,仅处理最富信息的图像块,使内存使用量随所选图像块序列长度显著减少。该方法与视觉神经科学发现一致,表明人类视觉系统通过选择性空间动态聚焦优化信息获取。在ImageNet-1k上的实验表明,SMT在仅使用有限数量图像块时,仍能实现具有竞争力的top-1准确率,同时显著降低内存消耗和计算复杂度。本工作将类似扫视的选择机制引入基于Transformer的视觉模型,为图像分析提供了一种高效替代方案,并为资源受限场景下的生物启发式架构提供了新见解。
原文摘要 · Abstract (English)
This paper presents the Sensorimotor Transformer (SMT), a vision model inspired by human saccadic eye movements that prioritize high-saliency regions in visual input to enhance computational efficiency and reduce memory consumption. Unlike traditional models that process all image patches uniformly, SMT identifies and selects the most salient patches based on intrinsic two-dimensional (i2D) features, such as corners and occlusions, which are known to convey high-information content and align with human fixation patterns. The SMT architecture uses this biological principle to leverage vision transformers to process only the most informative patches, allowing for a substantial reduction in memory usage that scales with the sequence length of selected patches. This approach aligns with visual neuroscience findings, suggesting that the human visual system optimizes information gathering through selective, spatially dynamic focus. Experimental evaluations on Imagenet-1k demonstrate that SMT achieves competitive top-1 accuracy while significantly reducing memory consumption and computational complexity, particularly when a limited number of patches is used. This work introduces a saccade-like selection mechanism into transformer-based vision models, offering an efficient alternative for image analysis and providing new insights into biologically motivated architectures for resource-constrained applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。