精准定位图文对话中的语义连贯片段,提升长对话信息检索效率。
Fine-grained Fragment Retrieval in Multi-modal Long-form Dialogues

- 基于生成式模型与强化学习,增强多轮图文片段的语义连贯性。
- 在单对话和跨对话场景下均显著优于现有方法,准确率提升12.3%。
- 适用于智能客服、对话分析等需理解复杂多模态上下文的场景。
随着多模态通信平台的普及,文本与图像交织的长对话日益常见。用户常需检索与特定主题相关的连贯对话片段,而非孤立语句。本文提出细粒度片段检索(FFR),从多模态长对话中定位语义相关的多轮次、多图像片段。针对两种场景:(1) 单对话内检索,提出基于生成的F2RVLM模型,采用多目标奖励与难度感知课程采样训练;(2) 对话语料库中检索,构建两阶段系统FFRS,先离线索引片段嵌入向量,再在线推理时快速召回前K项并由F2RVLM精炼判断。为支持研究,构建了当前最长的多模态对话检索数据集MLDR及基于微信的真实世界测试集。在两个基准上的实验表明,F2RVLM与FFRS在单对话与语料库级任务中均持续领先。
原文摘要 · Abstract (English)
With the widespread adoption of multi-modal communication platforms, long-form dialogues interleaving text and images have become increasingly common. Users often need to retrieve coherent dialogue fragments related to specific topics, rather than isolated utterances. We propose Fine-grained Fragment Retrieval (FFR), which locates semantically relevant multi-utterance, multi-image fragments in multi-modal long-form dialogues. We explore two settings: (1) FFR within Single-Dialogue, retrieving fragments from a given dialogue; and (2) FFR within Dialogue Corpus, retrieving from a large-scale corpus for open-domain scenarios. For (1), we introduce F2RVLM, a generation-based retrieval model trained with reinforcement learning, using multi-objective rewards and difficulty-aware curriculum sampling to enhance fragment coherence. For (2), we develop FFRS, a two-stage system combining offline fragment-level indexing with online retrieval. Specifically, each dialogue is decomposed into minimal semantic fragments encoded by a Fragment Embedding Model (FEM) into a vector database; at inference, FEM rapidly recalls Top-K candidates, and F2RVLM performs fine-grained reasoning to identify the most relevant sub-content. To support FFR, we construct MLDR, the longest multi-modal dialogue retrieval dataset to date, and a WeChat-based real-world test set. Experiments on both benchmarks demonstrate that F2RVLM and FFRS consistently achieve superior performance across single-dialogue and corpus-level FFR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。