MambaEye实现图像尺寸无关的视觉编码，支持任意分辨率输入。
MambaEye: A Size-Agnostic Visual Encoder with Causal Sequential Processing
- 采用因果顺序处理的Mamba2架构，仅单向扫描图像块。
- 在1536²分辨率下图像分类性能显著优于传统方法。
- 适合需要灵活适应不同分辨率的视觉任务应用。
尽管历经数十年发展，真正具备输入尺寸无关性的视觉编码器——人类视觉的核心特征——仍未能实现。本文提出新型因果序列编码器MambaEye，基于低复杂度且具有因果特性的纯Mamba2主干网络。与以往采用双向处理的Mamba视觉编码器不同，本方法严格使用单向序列处理，保留状态空间模型的固有因果性，使模型可在输入序列任意时刻生成预测。核心创新在于引入相对位移嵌入，编码连续图像块间的空间偏移，为平移不变性提供强归纳偏置，使模型天然适配任意图像分辨率和扫描模式。为此，我们设计了一种受扩散模型启发的损失函数，提供密集、分步监督，训练模型随视觉信息积累逐步建立置信度。实验表明，MambaEye在多种图像分辨率下表现稳健，尤其在ImageNet-1K分类任务中，1536²分辨率下表现优异。该方法在计算时间和内存上均保持与图像块数量线性增长的复杂度。
原文摘要 · Abstract (English)
Despite decades of progress, a truly input-size agnostic visual encoder-a fundamental characteristic of human vision-has remained elusive. We address this limitation by proposing \textbf{MambaEye}, a novel, causal sequential encoder that leverages the low complexity and causal-process based pure Mamba2 backbone. Unlike previous Mamba-based vision encoders that often employ bidirectional processing, our strictly unidirectional approach preserves the inherent causality of State Space Models, enabling the model to generate a prediction at any point in its input sequence. A core innovation is our use of relative move embedding, which encodes the spatial shift between consecutive patches, providing a strong inductive bias for translation invariance and making the model inherently adaptable to arbitrary image resolutions and scanning patterns. To achieve this, we introduce a novel diffusion-inspired loss function that provides dense, step-wise supervision, training the model to build confidence as it gathers more visual evidence. We demonstrate that MambaEye exhibits robust performance across a wide range of image resolutions, especially at higher resolutions such as $1536^2$ on the ImageNet-1K classification task. This feat is achieved while maintaining linear time and memory complexity relative to the number of patches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。