用点提示快速生成高质量图像区域编码,速度比现有方法快60倍。
REN: Fast and Efficient Region Encodings from Patch-Based Image Encoders
- 通过轻量跨注意力模块直接从点提示生成区域令牌,跳过耗时分割步骤。
- 在语义分割与检索任务中,性能优于原生编码器,且令牌更紧凑。
- 适用于多种图像编码器,适合需要高效视觉理解的实时应用。
我们提出区域编码网络(REN),一种基于点提示快速生成区域图像表示的高效模型。现有方法结合无类别分割器(如SAM)与基于图像块的编码器(如DINO)生成紧凑有效的区域表示,但因分割步骤导致计算开销高。REN通过轻量级模块直接生成区域令牌,实现60倍加速与35倍内存节省,同时提升令牌质量。其采用少量跨注意力块,以点提示为查询,图像块编码器特征为键值,生成对应提示对象的区域令牌。我们在DINO、DINOv2和OpenCLIP三个主流编码器上训练REN,并证明其可扩展至其他编码器而无需额外训练。在语义分割与检索任务中,REN持续优于原始编码器,在性能与紧凑性上均表现更优,且显著快于基于SAM的方法。尤其在挑战性的Ego4D VQ2D基准上达到当前最优结果,在Visual Haystacks单针挑战中超越专有大模型。代码与模型已开源:https://github.com/savya08/REN。
原文摘要 · Abstract (English)
We introduce the Region Encoder Network (REN), a fast and effective model for generating region-based image representations using point prompts. Recent methods combine class-agnostic segmenters (e.g., SAM) with patch-based image encoders (e.g., DINO) to produce compact and effective region representations, but they suffer from high computational cost due to the segmentation step. REN bypasses this bottleneck using a lightweight module that directly generates region tokens, enabling 60x faster token generation with 35x less memory, while also improving token quality. It uses a few cross-attention blocks that take point prompts as queries and features from a patch-based image encoder as keys and values to produce region tokens that correspond to the prompted objects. We train REN with three popular encoders-DINO, DINOv2, and OpenCLIP-and show that it can be extended to other encoders without dedicated training. We evaluate REN on semantic segmentation and retrieval tasks, where it consistently outperforms the original encoders in both performance and compactness, and matches or exceeds SAM-based region methods while being significantly faster. Notably, REN achieves state-of-the-art results on the challenging Ego4D VQ2D benchmark and outperforms proprietary LMMs on Visual Haystacks' single-needle challenge. Code and models are available at: https://github.com/savya08/REN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。