用现成模型生成带文本对齐的区域标记，实现精准图文理解。
TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Models
- 基于冻结图文模型与SAM2融合生成文本对齐区域令牌
- 在开放世界分割等任务上超越或媲美顶尖无训练方法
- 无需训练、适配多种模型，适合快速部署
图文模型在图像级任务中表现优异，但在细节视觉理解上存在不足。尽管这些模型具备强大的视觉-语言对齐能力，分割模型如SAM2却能提供精确的对象空间边界。为此，我们提出TextRegion——一种简单、高效且无需训练的框架，结合图文模型与SAM2的优势，生成强大的文本对齐区域令牌。这些令牌在保持开放词汇能力的同时，支持详细视觉理解，并可直接应用于开放世界语义分割、指代表达理解与定位等多种下游任务。我们在多个任务上进行了广泛评估，结果表明其性能持续优于或媲美当前最先进的无训练方法。此外，该框架兼容多种图文模型，具有高度实用性与可扩展性，未来更强模型出现时可无缝集成。代码已开源：https://github.com/avaxiao/TextRegion。
原文摘要 · Abstract (English)
Image-text models excel at image-level tasks but struggle with detailed visual understanding. While these models provide strong visual-language alignment, segmentation models like SAM2 offer precise spatial boundaries for objects. To this end, we propose TextRegion, a simple, effective, and training-free framework that combines the strengths of image-text models and SAM2 to generate powerful text-aligned region tokens. These tokens enable detailed visual understanding while preserving open-vocabulary capabilities. They can be directly applied to various downstream tasks, including open-world semantic segmentation, referring expression comprehension, and grounding. We conduct extensive evaluations and consistently achieve superior or competitive performance compared to state-of-the-art training-free methods. Additionally, our framework is compatible with many image-text models, making it highly practical and easily extensible as stronger models emerge. Code is available at: https://github.com/avaxiao/TextRegion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。