arXiv:2510.23603cs.CV2025-10被引 24

让大模型精准理解图片视频中任意区域的物体,支持细粒度定位与问答。

PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularity

  • 用自适应对象分词器生成自由区域的紧凑语义表示。
  • 在多个基准上以更少训练样本达到领先性能,轻量版效率提升显著。
  • 适合需要精细视觉理解的应用,如智能客服、医疗影像分析。

多模态大语言模型在开放世界视觉理解中展现出强大通用能力,但多数模型仍聚焦整体场景理解,忽视细粒度物体中心推理需求。本文提出PixelRefer,一种统一的区域级多模态大模型框架,支持图像与视频中用户指定区域的细粒度理解。受大语言模型注意力集中在物体级别标记的启发,我们设计了尺度自适应对象分词器(SAOT),从自由区域生成紧凑且语义丰富的对象表征。分析发现全局视觉标记主要在大模型早期层起作用,由此提出PixelRefer-Lite,通过对象中心注入模块预先融合全局上下文至对象标记,形成轻量级仅对象框架,在大幅降低计算开销的同时保持高语义保真度。为支持细粒度指令微调,我们构建了高质量的对象中心指令数据集PixelRefer-2.2M。大量实验表明,PixelRefer在较少训练样本下即实现领先性能,PixelRefer-Lite则在效率上表现突出且精度具竞争力。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) have demonstrated strong general-purpose capabilities in open-world visual comprehension. However, most existing MLLMs primarily focus on holistic, scene-level understanding, often overlooking the need for fine-grained, object-centric reasoning. In this paper, we present PixelRefer, a unified region-level MLLM framework that enables advanced fine-grained understanding over user-specified regions across both images and videos. Motivated by the observation that LLM attention predominantly focuses on object-level tokens, we propose a Scale-Adaptive Object Tokenizer (SAOT) to generate compact and semantically rich object representations from free-form regions. Our analysis reveals that global visual tokens contribute mainly in early LLM layers, inspiring the design of PixelRefer-Lite, an efficient variant that employs an Object-Centric Infusion module to pre-fuse global context into object tokens. This yields a lightweight Object-Only Framework that substantially reduces computational cost while maintaining high semantic fidelity. To facilitate fine-grained instruction tuning, we curate PixelRefer-2.2M, a high-quality object-centric instruction dataset. Extensive experiments across a range of benchmarks validate that PixelRefer achieves leading performance with fewer training samples, while PixelRefer-Lite offers competitive accuracy with notable gains in efficiency.

多模态细粒度理解对象定位轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。