arXiv:2604.18573cs.CV2026-04被引 2

T-REN通过文本对齐区域令牌,显著提升视觉语言对齐与长视频处理效率。

T-REN: Learning Text-Aligned Region Tokens Improves Dense Vision-Language Alignment and Scalability

论文配图:T-REN: Learning Text-Aligned Region Tokens Improves Dense Vision-Language Alignment and Scalability
图 1 · 摘自论文原文
  • 用轻量网络将视觉特征聚合成文本对齐的区域令牌
  • 在多个任务上性能提升超5.9个百分点,令牌数减少24倍以上
  • 适合需要高精度对齐和长视频处理的研究与应用

尽管近期取得进展,视觉-语言编码器仍存在两大核心问题:(1) 语言与密集视觉特征对齐弱,影响开放词汇语义分割等任务;(2) 细粒度视觉表示所需令牌数量高,限制了长视频的可扩展性。本文提出T-REN(文本对齐区域编码器网络),一种高效编码器,将视觉数据映射为一组紧凑的文本对齐区域级表示(区域令牌)。T-REN在冻结的视觉主干网络之上添加轻量级网络,训练其将每个语义区域内局部补丁特征池化为区域令牌,并与区域级文本标注对齐。相比视觉-语言主干仅增加3.7%参数,该设计大幅增强密集跨模态理解能力,同时将令牌数量降低数个数量级。具体而言,T-REN在ADE20K开放词汇分割任务上提升5.9 mIoU,COCO物体级图文检索召回率提升18.4%,Ego4D视频物体定位提升15.6%,VSPW视频场景解析提升17.6% mIoU,图像令牌数减少24倍以上,视频令牌数减少187倍以上。代码与模型已开源于https://github.com/savya08/T-REN。

原文摘要 · Abstract (English)

Despite recent progress, vision-language encoders struggle with two core limitations: (1) weak alignment between language and dense vision features, which hurts tasks like open-vocabulary semantic segmentation; and (2) high token counts for fine-grained visual representations, which limits scalability to long videos. This work addresses both limitations. We propose T-REN (Text-aligned Region Encoder Network), an efficient encoder that maps visual data to a compact set of text-aligned region-level representations (or region tokens). T-REN achieves this through a lightweight network added on top of a frozen vision backbone, trained to pool patch-level representations within each semantic region into region tokens and align them with region-level text annotations. With only 3.7% additional parameters compared to the vision-language backbone, this design yields substantially stronger dense cross-modal understanding while reducing the token count by orders of magnitude. Specifically, T-REN delivers +5.9 mIoU on ADE20K open-vocabulary segmentation, +18.4% recall on COCO object-level text-image retrieval, +15.6% recall on Ego4D video object localization, and +17.6% mIoU on VSPW video scene parsing, all while reducing token counts by more than 24x for images and 187x for videos compared to the patch-based vision-language backbone. The code and model are available at https://github.com/savya08/T-REN.

视觉语言对齐区域令牌视频理解高效编码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。