用记忆增强的视觉模型，速度提升56%且更省算力。
Token Turing Machines are Efficient Vision Models
- 引入过程令牌与记忆令牌，动态存取信息
- 图像分类准确率82.9%，推理速度比ViT-B快56%
- 适合对延迟敏感的视觉任务，如实时分割
我们提出视觉令牌图灵机（ViTTM），一种高效、低延迟、带记忆增强的视觉Transformer（ViT）。该方法基于神经图灵机和令牌图灵机，应用于非序列化计算机视觉任务如图像分类与分割。模型生成两类令牌：过程令牌和记忆令牌；过程令牌在编码器块中传递，并在每个块读写记忆令牌，实现信息存储与检索。通过保持过程令牌少于记忆令牌，显著降低推理时间而不损失精度。在ImageNet-1K上，当前最优的ViT-B模型中位延迟为529.5ms，准确率为81.0%；而我们的ViTTM-B模型延迟降至234.1ms（提速56%），仅需2.4倍少的浮点运算量（FLOPs），准确率达82.9%。在ADE20K语义分割任务中，ViT-B达到45.65mIoU，帧率为13.8 FPS；ViTTM-B模型取得45.17mIoU，帧率提升至26.8 FPS（+94%）。
原文摘要 · Abstract (English)
We propose Vision Token Turing Machines (ViTTM), an efficient, low-latency, memory-augmented Vision Transformer (ViT). Our approach builds on Neural Turing Machines and Token Turing Machines, which were applied to NLP and sequential visual understanding tasks. ViTTMs are designed for non-sequential computer vision tasks such as image classification and segmentation. Our model creates two sets of tokens: process tokens and memory tokens; process tokens pass through encoder blocks and read-write from memory tokens at each encoder block in the network, allowing them to store and retrieve information from memory. By ensuring that there are fewer process tokens than memory tokens, we are able to reduce the inference time of the network while maintaining its accuracy. On ImageNet-1K, the state-of-the-art ViT-B has median latency of 529.5ms and 81.0% accuracy, while our ViTTM-B is 56% faster (234.1ms), with 2.4 times fewer FLOPs, with an accuracy of 82.9%. On ADE20K semantic segmentation, ViT-B achieves 45.65mIoU at 13.8 frame-per-second (FPS) whereas our ViTTM-B model acheives a 45.17 mIoU with 26.8 FPS (+94%).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。