arXiv:2504.20734cs.CLcs.AI2025-04ACL被引 18

UniversalRAG支持跨模态跨粒度检索,让AI回答更全面准确。

UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities

  • 按查询模态动态选择对应知识库,避免跨模态检索偏差
  • 在10个多模态基准上超越单一模态与统一表示的基线模型
  • 适合需要融合图文视频等多源信息的复杂问答场景

检索增强生成(RAG)通过引入外部知识提升了模型回答的事实准确性。然而,现有方法大多仅限于文本语料库,尽管近期工作已将RAG扩展至图像、视频等模态,但通常仅针对单一模态语料库。现实中查询所需的知识类型多样,单一知识源难以覆盖。为此,我们提出UniversalRAG,一个支持任意模态与粒度间检索的通用RAG框架。受观察启发:将所有模态统一映射到单一聚合语料库的表征空间会导致模态差距,使检索偏向与查询同模态的内容,我们提出模态感知路由机制,动态识别最合适的模态专用语料库并进行定向检索,并通过理论分析验证其有效性。此外,我们对每种模态划分多个粒度层级,实现与查询复杂度匹配的精细化检索。我们在10个不同模态的基准上验证了UniversalRAG,结果表明其显著优于多种模态专用及统一表示的基线模型。

原文摘要 · Abstract (English)

Retrieval-Augmented Generation (RAG) has shown substantial promise in improving factual accuracy by grounding model responses with external knowledge relevant to queries. However, most existing approaches are limited to a text-only corpus, and while recent efforts have extended RAG to other modalities such as images and videos, they typically operate over a single modality-specific corpus. In contrast, real-world queries vary widely in the type of knowledge they require, which a single type of knowledge source cannot address. To address this, we introduce UniversalRAG, an any-to-any RAG framework designed to retrieve and integrate knowledge from heterogeneous sources with diverse modalities and granularities. Specifically, motivated by the observation that forcing all modalities into a unified representation space derived from a single aggregated corpus causes a modality gap, where the retrieval tends to favor items from the same modality as the query, we propose modality-aware routing, which dynamically identifies the most appropriate modality-specific corpus and performs targeted retrieval within it, and further justify its effectiveness with a theoretical analysis. Moreover, beyond modality, we organize each modality into multiple granularity levels, enabling fine-tuned retrieval tailored to the complexity and scope of the query. We validate UniversalRAG on 10 benchmarks of multiple modalities, showing its superiority over various modality-specific and unified baselines.

RAG多模态知识检索

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。