提出动态混合注意力机制,让交互分割在CPU上又快又准。
Inter2Former: Dynamic Hybrid Attention for Efficient High-Precision Interactive
- 根据图像区域动态分配计算量,只处理重点区域。
- 在多个数据集上达到最高精度,且在CPU上推理速度快。
- 适合需要高精度且资源受限的实时交互场景使用。
交互式分割(IS)通过用户提示分割目标区域,显著提升标注效率,广泛应用于实际场景。现有方法面临核心矛盾:密集令牌方法虽精度高、细节好,但在CPU上计算极慢;而通用分割模型(SAM)采用稀疏提示令牌实现快速推理,但牺牲了分割质量。本文提出Inter2Former,通过优化密集令牌计算分配,引入四项关键改进:首先,动态提示嵌入(DPE)仅处理感兴趣区域,避免背景令牌的额外开销;其次,动态混合注意力(DHA)利用历史分割掩码,对边界区域使用全注意力(O(N²)),非边界区域则采用高效的BSQ注意力(O(N));第三,混合专家模块(HMoE)在前馈网络中应用类似策略,并支持CPU优化并行处理;最后,动态局部上采样(DLU)作为DPE的逆操作,用轻量MLP定位物体,并仅在检测区域进行精细上采样。在高精度交互分割基准测试中,Inter2Former在CPU设备上实现当前最优性能,兼具高精度与高效性。
原文摘要 · Abstract (English)
Interactive segmentation (IS) improves annotation efficiency by segmenting target regions from user prompts, with widespread applications in real-world scenarios. Current approaches face a critical trade-off: dense-token methods achieve superior accuracy and detail preservation but suffer from prohibitively slow processing on CPU devices, while the Segment Anything Model (SAM) advances the field with sparse prompt tokens for fast inference but compromises segmentation quality. In this paper, we propose Inter2Former to address this challenge by optimizing computation allocation in dense-token processing, which introduces four key enhancements. First, we propose Dynamic Prompt Embedding (DPE) that adaptively processes only regions of interest while avoiding additional overhead from background tokens. Second, we introduce Dynamic Hybrid Attention (DHA), which leverages previous segmentation masks to route tokens through either full attention (O(N2)) for boundary regions or our proposed efficient BSQ attention (O(N)) for non-boundary regions. Third, we develop Hybrid Mixture of Experts (HMoE), which applies similar adaptive computation strategies in FFN modules with CPU-optimized parallel processing. Finally, we present Dynamic Local Upsampling (DLU), a reverse operation of DPE, which localizes objects with a lightweight MLP and performs fine-grained upsampling only in detected regions. Experimental results on high-precision IS benchmarks demonstrate that Inter2Former achieves SOTA performance with high efficiency on CPU devices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。