arXiv:2411.11706cs.CVcs.AI2024-11被引 58

让视觉语言模型一次理解多个用户概念,提升个性化交互能力。

MC-LLaVA: Multi-Concept Personalized Vision-Language Model

  • 用多概念指令微调,一步整合多个用户概念。
  • 在电影图像上构建多角色问答数据集,支持复杂场景理解。
  • 适合需要多概念识别的智能助手、个性化推荐等应用。

当前视觉语言模型(VLM)在视觉问答等任务中表现优异。为提升用户体验,近期研究探索了用户概念个性化,但主要聚焦单一概念,忽视多个概念共存与相互作用,限制了实际应用。本文提出MC-LLaVA,一种多概念个性化范式。通过多概念指令微调策略,可在单次训练中有效融合多个概念。为降低训练成本,提出基于视觉标记信息初始化概念标记的个性化文本提示。推理时引入个性化视觉提示,聚合位置图以增强识别与定位能力。此外,可选辅助损失进一步提升提示效果。为推动该方向研究,我们构建了一个高质量数据集,从电影中精心收集含多个角色和物体的图像,并手动标注多概念问答样本,具备优异多样性。全面实验表明,MC-LLaVA能生成出色的多概念个性化响应,为VLM成为更优用户助手铺平道路。代码与数据集将公开于 https://github.com/arctanxarc/MC-LLaVA。

原文摘要 · Abstract (English)

Current vision-language models (VLMs) show exceptional abilities across diverse tasks, such as visual question answering. To enhance user experience, recent studies have investigated VLM personalization to understand user-provided concepts. However, they mainly focus on single concepts, neglecting the existence and interplay of multiple concepts, which limits real-world applicability. This paper proposes MC-LLaVA, a multi-concept personalization paradigm. Specifically, MC-LLaVA employs a multi-concept instruction tuning strategy, effectively integrating multiple concepts in a single training step. To reduce the training costs, we propose a personalized textual prompt that uses visual token information to initialize concept tokens. Additionally, we introduce a personalized visual prompt during inference, aggregating location maps for enhanced recognition and grounding capabilities. To further push the performance upper bound, we incorporate an optional auxiliary loss, better enhancing the proposed personalized prompts. To decorate the VLM personalization research, we contribute a high-quality dataset. We carefully collect images with multiple characters and objects from movies and manually create question-answer samples for multi-concept scenarios, featuring superior diversity. Comprehensive experiments demonstrate that MC-LLaVA achieves impressive multi-concept personalized responses, paving the way for VLMs to become better user assistants. The code and dataset will be released at \href{https://github.com/arctanxarc/MC-LLaVA}{https://github.com/arctanxarc/MC-LLaVA}.

多概念个性化视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。