arXiv:2503.18623cs.CV2025-03ICCV被引 10

不训练即可个性化理解用户概念,靠检索与推理实现精准识别。

Training-Free Personalization via Retrieval and Reasoning on Fingerprints

  • 用视觉语言模型提取概念特征指纹,无需训练即可存储用户特定信息。
  • 通过思维链推理匹配最相似指纹,在视觉模糊场景下准确率提升显著。
  • 适合需要快速个性化、不愿频繁训练的用户或实时交互系统使用。

视觉语言模型在多模态推理方面取得显著进展,但仍难以理解用户特定概念。现有个性化方法依赖训练过程,成本高且体验差。本文首次探索无训练场景下的个性化问题,提出检索与推理个性化方法(R2P)。R2P利用视觉语言模型内部知识,先提取概念指纹——即定义概念类别的关键属性;查询时,通过思维链推理检索并评分最相似指纹。为降低幻觉风险,采用跨模态属性级验证:若评分不一致,则通过成对多模态匹配修正概念关联,直接比对检索到的指纹图像与查询内容。在两个公开基准和新提出的包含视觉模糊挑战的数据集PerVA上验证,R2P在各类下游任务中持续优于现有最优方法。代码将在录用后公开。

原文摘要 · Abstract (English)

Vision Language Models (VLMs) have lead to major improvements in multimodal reasoning, yet they still struggle to understand user-specific concepts. Existing personalization methods address this limitation but heavily rely on training procedures, that can be either costly or unpleasant to individual users. We depart from existing work, and for the first time explore the training-free setting in the context of personalization. We propose a novel method, Retrieval and Reasoning for Personalization (R2P), leveraging internal knowledge of VLMs. First, we leverage VLMs to extract the concept fingerprint, i.e., key attributes uniquely defining the concept within its semantic class. When a query arrives, the most similar fingerprints are retrieved and scored via chain-of-thought-reasoning. To reduce the risk of hallucinations, the scores are validated through cross-modal verification at the attribute level: in case of a discrepancy between the scores, R2P refines the concept association via pairwise multimodal matching, where the retrieved fingerprints and their images are directly compared with the query. We validate R2P on two publicly available benchmarks and a newly introduced dataset, Personal Concepts with Visual Ambiguity (PerVA), for concept identification highlighting challenges in visual ambiguity. R2P consistently outperforms state-of-the-art approaches on various downstream tasks across all benchmarks. Code will be available upon acceptance.

个性化零样本视觉语言模型指纹检索

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。