让多模态大模型记住用户信息并实时调用，生成个性化回答。
RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models
- 构建外部数据库存用户信息，通过多模态检索实时获取。
- 在图像描述、问答等任务中表现优异，支持无限视觉概念泛化。
- 支持实时修改用户信息，适合个性化助手开发场景。
大型语言模型（LLM）的发展显著提升了多模态大语言模型（MLLM）作为通用助手的能力，但缺乏用户特定知识仍限制其在日常生活中应用。本文提出检索增强型个性化框架（RAP），分三步将通用MLLM转化为个性化助手：(a) 记忆：设计键值数据库存储用户信息，如姓名、头像等；(b) 检索：用户对话时，通过多模态检索器从数据库中提取相关信息；(c) 生成：将查询与检索到的概念输入MLLM，生成个性化知识增强的回答。相比以往方法，RAP可通过更新外部数据库实现实时概念编辑。为提升生成质量与用户信息对齐，我们设计数据收集流程并构建专用数据集，用于个性化训练。基于该数据集，训练出一系列个性化多模态助手。预训练后的RAP-MLLM可在无需额外微调的情况下泛化至无限视觉概念。实验表明，模型在个性化图像描述、问答和视觉识别等任务中表现出色。代码、数据与模型已公开于https://hoar012.github.io/RAP-Project/。
原文摘要 · Abstract (English)
The development of large language models (LLMs) has significantly enhanced the capabilities of multimodal LLMs (MLLMs) as general assistants. However, lack of user-specific knowledge still restricts their application in human's daily life. In this paper, we introduce the Retrieval Augmented Personalization (RAP) framework for MLLMs' personalization. Starting from a general MLLM, we turn it into a personalized assistant in three steps. (a) Remember: We design a key-value database to store user-related information, e.g., user's name, avatar and other attributes. (b) Retrieve: When the user initiates a conversation, RAP will retrieve relevant information from the database using a multimodal retriever. (c) Generate: The input query and retrieved concepts' information are fed into MLLMs to generate personalized, knowledge-augmented responses. Unlike previous methods, RAP allows real-time concept editing via updating the external database. To further improve generation quality and alignment with user-specific information, we design a pipeline for data collection and create a specialized dataset for personalized training of MLLMs. Based on the dataset, we train a series of MLLMs as personalized multimodal assistants. By pretraining on large-scale dataset, RAP-MLLMs can generalize to infinite visual concepts without additional finetuning. Our models demonstrate outstanding flexibility and generation quality across a variety of tasks, such as personalized image captioning, question answering and visual recognition. The code, data and models are available at https://hoar012.github.io/RAP-Project/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。