让大视觉语言模型一键识别具体物体,提升对熟悉目标的辨识能力。
IIR-VLM: In-Context Instance-level Recognition for Large Vision-Language Models
- 引入专家模型作为辅助编码器,实现上下文内快速学习新实例
- 在多人重识别等任务上显著优于传统方法,支持零样本泛化
- 适合需要个性化视觉理解的应用场景,如人脸识别与宠物识别
实例级识别(ILR)旨在区分个体实例,以行人重识别为代表。尽管现代视觉语言模型(VLM)具备强大视觉感知能力,但在ILR任务上表现不佳,远逊于专用模型。现有方法多依赖特定数据集逐个训练,成本高且难以处理细粒度差异。本文提出IIR-VLM,通过集成预训练的ILR专家模型作为辅助视觉编码器,使VLM能在上下文中以单样本方式学习新实例,并实现实例感知的视觉理解。实验验证了其在现有实例个性化基准上的有效性,并在一项涵盖多种类别和难度的新基准上展现出优越的ILR性能,涉及人、人脸、宠物及通用物体。
原文摘要 · Abstract (English)
Instance-level recognition (ILR) concerns distinguishing individual instances from one another, with person re-identification as a prominent example. Despite the impressive visual perception capabilities of modern VLMs, we find their performance on ILR unsatisfactory, often dramatically underperforming domain-specific ILR models. This limitation hinders many practical application of VLMs, e.g. where recognizing familiar people and objects is crucial for effective visual understanding. Existing solutions typically learn to recognize instances one at a time using instance-specific datasets, which not only incur substantial data collection and training costs but also struggle with fine-grained discrimination. In this work, we propose IIR-VLM, a VLM enhanced for In-context Instance-level Recognition. We integrate pre-trained ILR expert models as auxiliary visual encoders to provide specialized features for learning diverse instances, which enables VLMs to learn new instances in-context in a one-shot manner. Further, IIR-VLM leverages this knowledge for instance-aware visual understanding. We validate IIR-VLM's efficacy on existing instance personalization benchmarks. Finally, we demonstrate its superior ILR performance on a challenging new benchmark, which assesses ILR capabilities across varying difficulty and diverse categories, with person, face, pet and general objects as the instances at task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。