让像素级多模态模型自动生成并验证区域描述,提升图文理解与分割能力。
PixVL: Self-Supervised Training of Pixel-Level MLLMs via a Unified Mask--Text Consistency Cycle

- 通过统一的掩码-文本一致性循环,实现无监督自训练。
- 在COCO、LVIS数据集上,分割与理解任务均显著提升。
- 适合需要高精度区域交互的视觉语言模型研究者。
近期研究开发了支持区域分割与区域理解的像素级多模态大语言模型(MLLM),将多模态交互从整图扩展到具体物体和区域。然而,这些方法面临两大挑战:一是高质量掩码-文本对稀缺,导致大量掩码标注缺乏语言监督;二是监督格式与学习信号密度差异引发分割与理解任务间的优化干扰。为此,我们提出PixVL,一种自监督后训练框架,引入统一的掩码-文本一致性循环,使像素级MLLM能生成并自验证区域描述,从无标签数据中学习。我们发现仅依赖几何重建的循环不可靠,因为重分割的IoU不能真实反映语义质量与指代充分性。因此,PixVL引入置信度感知的语义验证机制,正确选择目标区域时给予奖励,错误选择则赋零奖励。同时,利用时间分离的视频帧或几何变换图像视图进行跨视角验证,防止循环学习陷入位置和形状捷径。最后,采用质量耦合的双向学习策略,以最高奖励描述引导文本到掩码学习。该策略将区域理解与分割从竞争关系转变为互为生成与验证的关系。实验表明,PixVL在区域理解与分割任务上均有显著提升。
原文摘要 · Abstract (English)
Recent studies develop pixel-level multimodal large language models (MLLMs) that support both Region Segmentation and Region Understanding, extending multimodal interaction from whole images to specific objects and regions. However, these methods face two fundamental challenges. First, the scarcity of high-quality mask--text pairs leaves abundant mask annotations without corresponding language supervision. Second, discrepancies in supervision formats and learning-signal densities induce optimization interference between Region Segmentation and Region Understanding. To address these challenges, we propose PixVL, a self-supervised post-training framework that introduces a unified Mask--Text Consistency Cycle, enabling pixel-level MLLMs to generate and self-verify regional descriptions and learn from unlabeled data. We found that direct cycle based solely on geometric reconstruction is unreliable because re-segmentation IoU does not faithfully reflect the semantic quality and referring sufficiency. PixVL therefore introduces confuser-aware semantic verification, which uses the model's confidence when it correctly chooses the target among highly similar candidate regions, and assigns zero reward to an incorrect choice. Meanwhile, PixVL performs cross-view verification using temporally separated video frames or geometrically transformed image views, preventing cyclic learning from collapsing to positional and shape shortcuts. Finally, a quality-coupled bidirectional learning strategy uses the highest-reward description to guide Text-to-Mask learning. This strategy transforms Region Understanding and Region Segmentation from competing tasks into mutual generators and verifiers. Experiments demonstrate that PixVL improves both region understanding task and segmentation task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。