构建多模态长文档理解基准与检索感知微调框架，提升模型长文问答能力。
M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework
- 设计面向多模态长文档的检索感知微调方法
- 在851个样本上实现4.6%的答对率提升
- 适合需要处理超长图文混合文档的研究者
理解并回答长篇多模态文档中的问题在商业和实际应用中具有重要意义。然而，这类文档常包含数百页的文本、图表和表格，人工阅读耗时极长。为此，本文提出M-LongDoc基准，包含851个样本，涵盖近期且更长的文档，并要求开放性答案而非仅抽取式回答。同时，提出首个直接针对多模态长文档检索场景的微调框架。通过全自动构建的训练语料，可高效适配开源模型。实验表明，该方法相比基线模型在回答正确率上相对提升4.6%。数据、代码和模型已公开于https://multimodal-documents.github.io。
原文摘要 · Abstract (English)
The ability to understand and answer questions over documents can be useful in many business and practical applications. However, documents often contain lengthy and diverse multimodal contents such as texts, figures, and tables, which are very time-consuming for humans to read thoroughly. Hence, there is an urgent need to develop effective and automated methods to aid humans in this task. In this work, we introduce M-LongDoc, a benchmark of 851 samples, and an automated framework to evaluate the performance of large multimodal models. We further propose a retrieval-aware tuning approach for efficient and effective multimodal document reading. Compared to existing works, our benchmark consists of more recent and lengthy documents with hundreds of pages, while also requiring open-ended solutions and not just extractive answers. To our knowledge, our training framework is the first to directly address the retrieval setting for multimodal long documents. To enable tuning open-source models, we construct a training corpus in a fully automatic manner for the question-answering task over such documents. Experiments show that our tuning approach achieves a relative improvement of 4.6% for the correctness of model responses, compared to the baseline open-source models. Our data, code, and models are available at https://multimodal-documents.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。