arXiv:2510.12323cs.AI2025-10被引 24

让大模型能跨文本、图像、表格等多模态信息检索,提升复杂文档理解能力。

RAG-Anything: All-in-One RAG Framework

  • 将多模态内容视为关联知识实体,用双图结构统一建模跨模态关系与语义。
  • 在长文档上性能显著优于现有方法,尤其在跨模态证据融合任务中提升明显。
  • 适合需要处理混合媒体数据的智能问答、科研分析等场景,开源可用。

检索增强生成(RAG)已成为突破大模型静态训练限制的关键范式。然而,现有RAG能力与现实多模态知识环境存在显著脱节:现代知识库包含文本、图像、表格和数学表达等多种形式,而现有框架仅支持文本,导致处理多模态文档时出现根本性缺失。本文提出RAG-Anything,一个统一的多模态RAG框架,实现对所有模态的全面知识检索。该方法将多模态内容重构为相互关联的知识实体,引入双图构建机制,统一捕捉跨模态关系与文本语义。开发了跨模态混合检索策略,结合结构化知识导航与语义匹配,实现异构内容的有效推理,使相关证据跨越多种模态得以联合利用。RAG-Anything在挑战性多模态基准测试中表现优异,显著超越当前最优方法,在长文档任务中优势尤为突出。该框架确立了多模态知识访问的新范式,消除了现有系统架构碎片化问题。代码已开源:https://github.com/HKUDS/RAG-Anything。

原文摘要 · Abstract (English)

Retrieval-Augmented Generation (RAG) has emerged as a fundamental paradigm for expanding Large Language Models beyond their static training limitations. However, a critical misalignment exists between current RAG capabilities and real-world information environments. Modern knowledge repositories are inherently multimodal, containing rich combinations of textual content, visual elements, structured tables, and mathematical expressions. Yet existing RAG frameworks are limited to textual content, creating fundamental gaps when processing multimodal documents. We present RAG-Anything, a unified framework that enables comprehensive knowledge retrieval across all modalities. Our approach reconceptualizes multimodal content as interconnected knowledge entities rather than isolated data types. The framework introduces dual-graph construction to capture both cross-modal relationships and textual semantics within a unified representation. We develop cross-modal hybrid retrieval that combines structural knowledge navigation with semantic matching. This enables effective reasoning over heterogeneous content where relevant evidence spans multiple modalities. RAG-Anything demonstrates superior performance on challenging multimodal benchmarks, achieving significant improvements over state-of-the-art methods. Performance gains become particularly pronounced on long documents where traditional approaches fail. Our framework establishes a new paradigm for multimodal knowledge access, eliminating the architectural fragmentation that constrains current systems. Our framework is open-sourced at: https://github.com/HKUDS/RAG-Anything.

多模态RAG知识检索大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。