提出F2RVLM模型,精准定位长对话中跨模态的细粒度片段。
F2RVLM: Boosting Fine-grained Fragment Retrieval for Multi-Modal Long-form Dialogue with Vision Language Model
- 分两阶段训练:监督微调注入检索知识,强化学习优化语义精度与连贯性。
- 在平均75.38轮的真实微信对话上,显著提升碎片化内容召回准确率。
- 引入难度感知课程采样,增强复杂长对话中的推理能力,适合多模态交互研究者。
传统对话检索仅选取最近的语句或图像,难以满足用户对长对话中分散的语义连贯内容的回溯需求。为此,我们定义了细粒度片段检索(FFR)任务,要求模型从多模态长对话中定位包含语句与图像的相关片段。为此构建了迄今最长的多模态对话检索数据集MLDR,平均每条对话25.45轮,自然覆盖三个不同主题。为评估真实场景泛化能力,我们采集并标注了一个基于微信的真实多模态对话测试集,平均75.38轮。基于此,我们发现现有生成式视觉语言模型(VLMs)在FFR任务中常返回语义不连贯的片段。尽管其在图文生成任务上表现优异,但缺乏确保片段内部一致性的显式监督。为此,我们提出F2RVLM,采用两阶段范式:(1) 监督微调注入片段级检索知识;(2) 基于GRPO的强化学习,结合多目标奖励(语义精确性、相关性、上下文连贯性)。为应对片段内复杂度差异(从密集到稀疏分布),我们引入难度感知课程采样,按模型预测难度排序并逐步引入更难样本,提升长多轮对话中的推理能力。F2RVLM在域内与真实域设置下均优于主流VLM,展现卓越检索性能。
原文摘要 · Abstract (English)
Traditional dialogue retrieval aims to select the most appropriate utterance or image from recent dialogue history. However, they often fail to meet users' actual needs for revisiting semantically coherent content scattered across long-form conversations. To fill this gap, we define the Fine-grained Fragment Retrieval (FFR) task, requiring models to locate query-relevant fragments, comprising both utterances and images, from multimodal long-form dialogues. As a foundation for FFR, we construct MLDR, the longest-turn multimodal dialogue retrieval dataset to date, averaging 25.45 turns per dialogue, with each naturally spanning three distinct topics. To evaluate generalization in real-world scenarios, we curate and annotate a WeChat-based test set comprising real-world multimodal dialogues with an average of 75.38 turns. Building on these resources, we explore existing generation-based Vision-Language Models (VLMs) on FFR and observe that they often retrieve incoherent utterance-image fragments. While optimized for generating responses from visual-textual inputs, these models lack explicit supervision to ensure semantic coherence within retrieved fragments. To this end, we propose F2RVLM, a generative retrieval model trained in a two-stage paradigm: (1) supervised fine-tuning to inject fragment-level retrieval knowledge, and (2) GRPO-based reinforcement learning with multi-objective rewards promoting semantic precision, relevance, and contextual coherence. To handle varying intra-fragment complexity, from locally dense to sparsely distributed, we introduce difficulty-aware curriculum sampling that ranks training instances by model-predicted difficulty and gradually exposes the model to harder samples. This boosts reasoning ability in long, multi-turn contexts. F2RVLM outperforms popular VLMs in both in-domain and real-domain settings, demonstrating superior retrieval performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。