无需训练即可理解复杂文档，通过伪目录引导的检索增强生成。
Zero-Shot Document Understanding using Pseudo Table of Contents-Guided Retrieval-Augmented Generation
- 用提示工程生成文档层级伪目录，结构化多模态内容。
- 处理平均49.4页、20971词元文档，查询延迟降低45%至2.12秒。
- 适合处理长文档、无标注数据场景，如学术论文或报告分析。
由于结构不一致和训练数据有限，理解复杂多模态文档仍具挑战。我们提出无需训练的文档理解系统DocsRay，结合伪目录生成与分层检索增强生成（RAG）。该方法利用多模态大模型的原生能力，无缝处理含文本、图像、图表、表格等元素的文档，无需专用模型或额外训练。DocsRay框架融合三项关键技术：(1) 基于提示的语义结构模块，生成层级伪目录；(2) 零样本多模态分析，将多样文档元素转化为统一文本表示；(3) 高效两阶段分层检索系统，将检索复杂度从$O(N)$降至$O(S + k_1 \cdot N_s)$。在平均49.4页、20,971文本标记的文档上测试，查询延迟由3.89秒降至2.12秒，效率提升45%。在MMLongBench-Doc基准上，DocsRay-Pro准确率达64.7%，显著超越此前最先进水平。
原文摘要 · Abstract (English)
Understanding complex multimodal documents remains challenging due to their structural inconsistencies and limited training data availability. We introduce \textit{DocsRay}, a training-free document understanding system that integrates pseudo Table of Contents (TOC) generation with hierarchical Retrieval-Augmented Generation (RAG). Our approach leverages multimodal Large Language Models' (LLMs) native capabilities to seamlessly process documents containing diverse elements such as text, images, charts, and tables without requiring specialized models or additional training. DocsRay's framework synergistically combines three key techniques: (1) a semantic structuring module using prompt-based LLM interactions to generate a hierarchical pseudo-TOC, (2) zero-shot multimodal analysis that converts diverse document elements into unified, text-centric representations using the inherent capabilities of multimodal LLMs, and (3) an efficient two-stage hierarchical retrieval system that reduces retrieval complexity from $O(N)$ to $O(S + k_1 \cdot N_s)$. Evaluated on documents averaging 49.4 pages and 20,971 textual tokens, DocsRay reduced query latency from 3.89 to 2.12 seconds, achieving a 45% efficiency improvement. On the MMLongBench-Doc benchmark, DocsRay-Pro attains an accuracy of 64.7%, substantially surpassing previous state-of-the-art results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。