让AI理解用户个性化知识间关系，提升推理能力。
ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant
- 用知识图谱+思维链问答对，让模型学会连接用户专属概念
- 在120个个性化知识集上测试，推理准确率显著优于基线
- 适合需要个性化知识推理的智能助手、教育类应用
多模态大语言模型在跨模态任务中表现强劲，近期个性化多模态模型能识别用户特定概念并生成上下文描述。然而，现有方法多聚焦孤立概念，缺乏关系训练数据，忽视个性化概念间的关联，且评估集中于识别或描述任务。为此，我们构建了包含120个个性化知识集的ReGraP数据集，每组包含图像、知识图谱（KG）及思维链问答对（CoT QA）。基于此，我们提出ReGraP-LLaVA：一种融合知识图谱与思维链问答的个性化多模态大模型，通过软/硬图提示将结构化关系知识对齐至模型语义空间。我们进一步建立ReGraP基准，涵盖选择题、填空题、判断题及开放/封闭式描述题，用于评估个性化关系推理与知识连接能力。实验表明，ReGraP-LLaVA有效学习个性化知识并实现关系推理，在多个任务上超越对比基线。代码与数据见：https://github.com/xyfyyds/ReGraP
原文摘要 · Abstract (English)
Multimodal Large Language Models have shown strong performance across multimodal tasks, and recent personalized MLLMs can recognize user-specific concepts and generate contextual captions. However, existing personalized MLLMs mainly focus on isolated concepts, often lacking relational training data, neglecting connections among personalized concepts, and evaluating mostly on recognition or captioning. To address these limitations, we introduce ReGraP, a dataset of 120 personalized knowledge sets, each containing images, knowledge graphs, and Chain-of-Thought Question-Answering pairs. Based on ReGraP, we propose Reasoning enabled Graph-based Personalized Large Language and Vision Assistant ReGraP-LLaVA, a personalized MLLM that incorporates KGs and CoT QA pairs through soft and/or hard graph prompting to align structured relational knowledge with the model's semantic space. We further establish the ReGraP Benchmark, covering multiple-choice, fill-in-the-blank, true/false, and descriptive questions in both open- and closed-ended settings, to evaluate personalized relational reasoning and knowledge-connection capabilities. Experimental results show that ReGraP-LLaVA effectively learns personalized knowledge and performs relational reasoning, achieving the best overall performance among competitive baselines. Code and data are available at: https://github.com/xyfyyds/ReGraP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。