arXiv:2510.03853cs.CV2025-10中稿 · ICML被引 14

用可学习的中间层替代固定最后一层,让文本提示更精准定位图像区域。

UGround: Towards Unified Visual Grounding with Unrolled Transformers

  • 通过随机跳过机制动态选择变压器中间层,实现灵活的特征融合。
  • 在多类视觉定位任务上均超越传统方法,最高提升12.3%
  • 适合需要精确空间定位的多模态任务,如细粒度图像分割。

我们提出 UGround,一种统一的视觉定位范式,通过展开的 Transformer 动态选择中间层作为「掩码即提示」,突破了以往使用固定最后隐藏层作为「<SEG>即提示」的局限。该范式解决了两大问题:一是固定层导致误差逐层累积且无法修正;二是<SEG>提示缺乏显式空间信息(如坐标)。核心是策略提示掩码(Policy-Prompted Masking),包含随机跳过连接(SSC)与掩码即提示(MasP)。SSC 以强化学习策略实现<SEG> token 在展开层间的随机滑动,动态决定与视觉模型(如 SAM)的跳跃连接层。选定层后,MasP 利用<SEG>与图像标记的相似性图生成软对数掩码,引导 SAM 生成分割掩码,激活区域提供明确空间线索。为验证有效性,首次在单一框架内统一了从传统指代表达分割到新提出的推理分割、单目标到多目标、正例到假前提(空目标)的多种任务。所有代码与模型已开源于 https://github.com/rui-qian/UGround。

原文摘要 · Abstract (English)

We present UGround, a \textbf{U}nified visual \textbf{Ground}ing paradigm that dynamically selects intermediate layers across \textbf{U}nrolled transformers as ``mask as prompt,'' diverging from the prevailing pipeline that leverages the fixed last hidden layer as ``\texttt{<SEG>} as prompt.'' UGround addresses two primary challenges posed by the prevailing paradigm: (1) its reliance on the fixed last hidden layer, which sequentially amplifies cumulative errors arising from layer-by-layer propagation without intermediate correction, and (2) its use of \texttt{<SEG>} as a prompt, which implicitly projects textual embeddings into visual space without explicit spatial cues (e.g., coordinates). Central to UGround is Policy-Prompted Masking, which comprises two key components: Stochastic Skip Connection (SSC) and Mask as Prompt (MasP). SSC is a reinforcement learning policy that, via stochastic sampling, allows each \texttt{<SEG>} token to slide across unrolled transformer layers, enabling dynamic layer selection at which it connects to the vision model (e.g., SAM) in a skip-connection fashion. Given the selected hidden layer, MasP uses the similarity map derived from the \texttt{<SEG>} token and image tokens as a soft logit mask to prompt SAM for mask generation, offering explicit spatial cues through its activation regions. To validate the effectiveness of UGround, we, for the first time, have unified visual grounding within a single framework from an attribute perspective, spanning from traditional refer expression segmentation to newly proposed reasoning segmentation, single-target to multi-target, positive query to false premise (empty target). All code and models are publicly available at https://github.com/rui-qian/UGround.

视觉定位Transformer多模态分割

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。