通过跨层融合相似视觉令牌，提升Vision Mamba训练效率。
Famba-V: Fast Vision Mamba with Cross-Layer Token Fusion
- 跨层策略识别并融合不同层的相似令牌，避免盲目统一融合。
- 在CIFAR-100上缩短训练时间，降低峰值显存占用。
- 适合追求高效训练的视觉Mamba模型研究与应用者。
Mamba及Vision Mamba（Vim）模型展现了作为Transformer替代方案的潜力。本文提出Fast Mamba for Vision（Famba-V），一种基于跨层令牌融合的训练效率增强技术。其核心思想是通过一系列跨层策略，识别并融合不同Vim层间的相似令牌，而非像现有方法那样对所有层统一进行令牌融合。我们在CIFAR-100数据集上评估了Famba-V的性能，结果表明，Famba-V可有效提升Vim模型的训练效率，显著减少训练时间与训练过程中的峰值内存使用。此外，所提出的跨层策略使Famba-V实现了更优的精度-效率权衡。这些结果共同证明Famba-V是Vim模型极具前景的效率优化技术。
原文摘要 · Abstract (English)
Mamba and Vision Mamba (Vim) models have shown their potential as an alternative to methods based on Transformer architecture. This work introduces Fast Mamba for Vision (Famba-V), a cross-layer token fusion technique to enhance the training efficiency of Vim models. The key idea of Famba-V is to identify and fuse similar tokens across different Vim layers based on a suit of cross-layer strategies instead of simply applying token fusion uniformly across all the layers that existing works propose. We evaluate the performance of Famba-V on CIFAR-100. Our results show that Famba-V is able to enhance the training efficiency of Vim models by reducing both training time and peak memory usage during training. Moreover, the proposed cross-layer strategies allow Famba-V to deliver superior accuracy-efficiency trade-offs. These results all together demonstrate Famba-V as a promising efficiency enhancement technique for Vim models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。