用聚合模型实现高效零样本语义分割，精度更高且更快更省资源。
RADSeg: Unleashing Parameter and Compute Efficient Zero-Shot Open-Vocabulary Segmentation Using Agglomerative Models
- 基于RADIO模型设计递归注意力与全局聚合机制，提升分割精度。
- 在ViT基础类上实现6%-30%的mIoU提升，推理速度加快3.95倍。
- 仅106M参数就超越千兆级模型组合，适合部署在资源受限场景。
开放词汇语义分割（OVSS）支撑众多需要泛化语义理解的视觉与机器人任务。现有方法或依赖有限分割训练数据，影响泛化能力；或采用零样本启发式策略应用于视觉语言模型（如CLIP），而高性能方案常需组合多个大模型，导致计算与内存开销巨大。本文首次系统研究了被忽视的聚合型视觉基础模型RADIO在零样本OVSS中的潜力，通过自相关递归注意力、自相关全局聚合及高效RADIO SAM掩码精修，提出RADSeg。该方法在保持高精度的同时，实现6-30% mIoU提升，推理速度提高3.95倍，参数量减少2.5倍。令人惊讶的是，仅106M参数的RADSeg-base性能超过此前850-1350M的大模型组合，在保持领先准确率的同时显著降低计算与内存成本。
原文摘要 · Abstract (English)
Open-vocabulary semantic segmentation (OVSS) underpins many vision and robotics tasks that require generalizable semantic understanding. Existing approaches either rely on limited segmentation training data, which hinders generalization, or apply zero-shot heuristics to vision-language models (e.g CLIP), while the most competitive approaches combine multiple models to improve performance at the cost of high computational and memory demands. In this work, we leverage an overlooked agglomerative vision foundation model, RADIO, to improve zero-shot OVSS along three key axes simultaneously: mIoU, latency, and parameter efficiency. We present the first comprehensive study of RADIO for zero-shot OVSS and enhance its performance through self-correlating recursive attention, self-correlating global aggregation, and computationally efficient RADIO SAM mask refinement. Our approach, RADSeg, achieves 6-30% mIoU improvement in the base ViT class while being 3.95x faster and using 2.5x fewer parameters. Surprisingly, RADSeg-base (106M) outperforms previous combinations of huge vision models (850-1350M) in mIoU, achieving state-of-the-art accuracy with substantially lower computational and memory cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。