LIRA通过局部视觉耦合提升多模态模型的分割与理解精度。
LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistance
- 用语义增强特征提取器融合语义与像素特征，提升分割准确率。
- 通过自回归生成局部描述，减少幻觉并改善细粒度理解。
- 发现分割精度与<seg>标记潜在语义相关，适合视觉理解研究者。
大型多模态模型（LMMs）在分割与理解任务中展现出潜力，但仍面临分割不准和理解幻觉的问题，主要源于弱视觉理解与缺乏细粒度感知。为此，我们提出LIRA框架，利用视觉理解与分割间的互补关系，包含两个核心组件：(1) 语义增强特征提取器（SEFE）通过融合语义与像素级特征，提升对象属性推理能力，实现更精准的分割；(2) 局部视觉耦合（ILVC）在分割掩码基础上自回归生成局部描述，提供细粒度监督以缓解幻觉。此外，我们发现物体分割精度与<seg>标记的潜在语义相关性正向显著。为量化此关系及模型的语义推断潜力，我们构建了属性评估数据集AttrEval。实验表明，LIRA在分割与理解任务上均达到当前最优性能。代码将发布于https://github.com/echo840/LIRA。
原文摘要 · Abstract (English)
While large multi-modal models (LMMs) demonstrate promising capabilities in segmentation and comprehension, they still struggle with two limitations: inaccurate segmentation and hallucinated comprehension. These challenges stem primarily from constraints in weak visual comprehension and a lack of fine-grained perception. To alleviate these limitations, we propose LIRA, a framework that capitalizes on the complementary relationship between visual comprehension and segmentation via two key components: (1) Semantic-Enhanced Feature Extractor (SEFE) improves object attribute inference by fusing semantic and pixel-level features, leading to more accurate segmentation; (2) Interleaved Local Visual Coupling (ILVC) autoregressively generates local descriptions after extracting local features based on segmentation masks, offering fine-grained supervision to mitigate hallucinations. Furthermore, we find that the precision of object segmentation is positively correlated with the latent related semantics of the <seg> token. To quantify this relationship and the model's potential semantic inferring ability, we introduce the Attributes Evaluation (AttrEval) dataset. Our experiments show that LIRA achieves state-of-the-art performance in both segmentation and comprehension tasks. Code will be available at https://github.com/echo840/LIRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。