用大模型实现跨图像物体部件的精准分割，提升多图理解能力。
CALICO: Part-Focused Semantic Co-Segmentation with Large Vision-Language Models
- 设计对应提取模块识别图像间部件级语义对应关系
- 仅微调0.3%参数即在240万样本上表现优异
- 适合需要细粒度跨图分析的研究者与应用
大型视觉语言模型（LVLM）虽能通过文本提示对单图生成分割掩码，但在跨图像的语义分割推理上仍面临挑战，尤其在物体部件等细粒度层面。本文提出新任务——面向部件的语义共分割，旨在识别并分割多张图像中的共同物体及共有与独有部件。为此，我们提出CALICO，首个专为多图像部件级推理分割设计的LVLM。其包含两个核心组件：新颖的对应提取模块用于识别部件级语义对应，以及对应适应模块将该信息高效嵌入LVLM以支持多图像理解。为支持训练与评估，我们构建了MixedParts数据集，涵盖约44,000张图像、约240万样本，覆盖多样化的物体与部件类别。实验表明，仅微调0.3%参数的CALICO，在此挑战性任务上表现强劲。
原文摘要 · Abstract (English)
Recent advances in Large Vision-Language Models (LVLMs) have enabled general-purpose vision tasks through visual instruction tuning. While existing LVLMs can generate segmentation masks from text prompts for single images, they struggle with segmentation-grounded reasoning across images, especially at finer granularities such as object parts. In this paper, we introduce the new task of part-focused semantic co-segmentation, which involves identifying and segmenting common objects, as well as common and unique object parts across images. To address this task, we present CALICO, the first LVLM designed for multi-image part-level reasoning segmentation. CALICO features two key components, a novel Correspondence Extraction Module that identifies semantic part-level correspondences, and Correspondence Adaptation Modules that embed this information into the LVLM to facilitate multi-image understanding in a parameter-efficient manner. To support training and evaluation, we curate MixedParts, a large-scale multi-image segmentation dataset containing $\sim$2.4M samples across $\sim$44K images spanning diverse object and part categories. Experimental results demonstrate that CALICO, with just 0.3% of its parameters finetuned, achieves strong performance on this challenging task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。