无需共现训练即可零样本组合个性化视觉语言模型。
Gate-and-Merge: Zero-shot Compositional Personalization of Vision Language Models
- 每个概念独立学习为轻量LoRA适配器，保持概念解耦。
- 推理时直接在权重空间合并适配器，支持任意组合。
- 门控机制筛选有效模块，防止干扰，保持概念清晰。
本文研究视觉语言模型（VLMs）的组合个性化问题，即在测试时需联合识别或描述多个用户自定义概念。提出一种零样本框架Gate-and-Merge，无需共现训练即可实现组合个性化。个性化阶段，每个概念以轻量级LoRA适配器形式独立学习，并关联概念标记，基础模型保持不变，概念间相互解耦。推理时，通过直接在权重空间合并特定概念的LoRA更新实现组合。为抑制无关激活和干扰，引入门控机制，根据文本与视觉线索选择贡献预测的模块。进一步通过仅融合最显著且一致的更新，稳定组合效果，保持各概念身份。定量与定性分析表明，在单概念及组合设置下，多种个性化任务均取得持续性能提升。
原文摘要 · Abstract (English)
This paper tackles compositional personalization of vision-language models (VLMs). In this problem, multiple user-defined concepts must be recognized or described jointly at test time. We introduce Gate-and-Merge, a zero-shot framework that enables compositional personalization without the need for co-occurrence training. During personalization, each concept is learned independently as a lightweight LoRA adapter, paired with a concept token. The base model remains unchanged and concepts are kept disentangled. At inference, we enable composition by merging concept-specific LoRA updates directly in weight space. To suppress irrelevant activations and prevent interference, a gating mechanism is employed to estimate textual and visual cues and select only the modules that contribute to the prediction. We further stabilize composition by combining only the most meaningful and mutually consistent updates, helping preserve each concept's identity. Our quantitative and qualitative analyses show consistent gains in performance across multiple personalization tasks in both single-concept and compositional settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。