通过聚类合并相似图像块，提升分割模型效率与速度。
ClustViT: Clustering-based Token Merging for Semantic Segmentation
- 用可学习聚类模块动态合并相似视觉令牌。
- 在三组数据集上减少2.18倍计算量，推理快1.64倍。
- 适合对实时性要求高的机器人视觉系统使用。
视觉变换器在多种场景下可实现高精度和强泛化能力，但其二次注意力复杂度限制了在真实机器人系统中的实际应用。近期工作尝试根据图像复杂度动态合并令牌，但在密集预测任务中效果不佳。本文提出ClustViT，扩展了视觉变换器主干网络以应对语义分割任务。架构中，一个可训练的聚类模块基于分割掩码生成的伪聚类，沿网络路径合并相似令牌；随后，重建模块恢复下游头所需的细粒度细节。该方法在三个不同数据集上实现最多2.18倍的GFLOPs减少和1.64倍的推理加速，同时保持相当的分割精度。代码与模型将公开发布。
原文摘要 · Abstract (English)
Vision Transformers can achieve high accuracy and strong generalization across various contexts, but their practical applicability on real-world robotic systems is limited due to their quadratic attention complexity. Recent works have focused on dynamically merging tokens according to the image complexity. Token merging works well for classification but is less suited to dense prediction. We propose ClustViT, where we expand upon the Vision Transformer (ViT) backbone and address semantic segmentation. Within our architecture, a trainable Cluster module merges similar tokens along the network guided by pseudo-clusters from segmentation masks. Subsequently, a Regenerator module restores fine details for downstream heads. Our approach achieves up to 2.18x fewer GFLOPs and 1.64x faster inference on three different datasets, with comparable segmentation accuracy. Our code and models will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。