iFormer融合卷积与注意力，移动端实现高精度低延迟图像识别。
iFormer: Integrating ConvNet and Transformer for Mobile Application
- 用轻量卷积+动态调制注意力，兼顾局部特征与全局建模。
- iPhone 13上达80.4%图像识别准确率，延迟仅1.10毫秒。
- 适合移动端高分辨率视觉任务，如检测、分割等场景。
我们提出一种新型移动混合视觉网络iFormer，专注于优化移动端的延迟与精度。iFormer有效结合卷积网络的快速局部表征能力与自注意力的高效全局建模能力。局部交互通过将标准卷积网络ConvNeXt改造为更轻量的移动端结构实现。新提出的移动调制注意力（mobile modulation attention）移除了内存密集型的多头注意力（MHA）操作，采用高效调制机制提升动态全局表征能力。大量实验表明，iFormer在多种任务上优于现有轻量级网络。尤其在ImageNet-1k上达到80.4%的Top-1准确率，iPhone 13上延迟仅1.10毫秒，超越相似延迟约束下的MobileNetV4。此外，在下游任务如COCO目标检测、实例分割和ADE20k语义分割中也表现显著提升，同时保持移动端对高分辨率输入的低延迟处理能力。
原文摘要 · Abstract (English)
We present a new family of mobile hybrid vision networks, called iFormer, with a focus on optimizing latency and accuracy on mobile applications. iFormer effectively integrates the fast local representation capacity of convolution with the efficient global modeling ability of self-attention. The local interactions are derived from transforming a standard convolutional network, \textit{i.e.}, ConvNeXt, to design a more lightweight mobile network. Our newly introduced mobile modulation attention removes memory-intensive operations in MHA and employs an efficient modulation mechanism to boost dynamic global representational capacity. We conduct comprehensive experiments demonstrating that iFormer outperforms existing lightweight networks across various tasks. Notably, iFormer achieves an impressive Top-1 accuracy of 80.4\% on ImageNet-1k with a latency of only 1.10 ms on an iPhone 13, surpassing the recently proposed MobileNetV4 under similar latency constraints. Additionally, our method shows significant improvements in downstream tasks, including COCO object detection, instance segmentation, and ADE20k semantic segmentation, while still maintaining low latency on mobile devices for high-resolution inputs in these scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。