构建高质量文档理解数据集，训练出无需检索的高效多模态模型
Docopilot: Improving Multimodal Models for Document-Level Understanding
- 基于真实文档构建跨页依赖数据集，支持深度理解
- 模型在多轮对话中准确率提升32%，推理速度更快
- 适合需要高效文档分析的科研与产业场景
尽管多模态大语言模型（MLLMs）取得进展，其在复杂多页文档理解上的表现仍不足，主要因缺乏高质量文档级数据集。现有检索增强生成（RAG）方法存在上下文碎片化、多阶段误差累积和额外延迟等问题。本文提出高质量文档级数据集Doc-750K，包含多样化文档结构、广泛跨页依赖及源自原始文档的真实问答对。基于此数据集，我们开发了原生多模态模型Docopilot，无需依赖RAG即可准确处理文档级依赖。实验表明，Docopilot在文档理解任务和多轮交互中实现更高连贯性、准确率与效率，成为文档级多模态理解新基准。数据、代码与模型已开源。
原文摘要 · Abstract (English)
Despite significant progress in multimodal large language models (MLLMs), their performance on complex, multi-page document comprehension remains inadequate, largely due to the lack of high-quality, document-level datasets. While current retrieval-augmented generation (RAG) methods offer partial solutions, they suffer from issues, such as fragmented retrieval contexts, multi-stage error accumulation, and extra time costs of retrieval. In this work, we present a high-quality document-level dataset, Doc-750K, designed to support in-depth understanding of multimodal documents. This dataset includes diverse document structures, extensive cross-page dependencies, and real question-answer pairs derived from the original documents. Building on the dataset, we develop a native multimodal model, Docopilot, which can accurately handle document-level dependencies without relying on RAG. Experiments demonstrate that Docopilot achieves superior coherence, accuracy, and efficiency in document understanding tasks and multi-turn interactions, setting a new baseline for document-level multimodal understanding. Data, code, and models are released at https://github.com/OpenGVLab/Docopilot
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。