不依赖OCR的文档理解新框架，提升多尺寸文本处理效率
Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding
- 用分层视觉特征聚合减少大模型输入令牌数
- 在保持精度前提下降低计算成本，适配不同大小文档
- 新增位置预测任务，防止大模型因容量限制截断文本
我们提出一种基于预训练多模态大语言模型（MLLMs）的新型无OCR文档理解框架。该方法采用多尺度视觉特征，有效应对文档图像中不同字号的问题。为缓解多尺度视觉输入带来的高计算开销，我们设计了分层视觉特征聚合（HVFA）模块，显著减少输入大模型的令牌数量。通过带有交叉注意力池化的特征金字塔，该方法在信息损失与计算效率间取得良好平衡，且不受文档图像尺寸变化影响。此外，我们引入一种新的指令微调任务，通过学习输入文本的相对位置，增强模型的文本阅读能力，从而降低大模型因容量限制导致的文本截断风险。大量实验验证了该方法在多种文档理解任务中的有效性。
原文摘要 · Abstract (English)
We present a novel OCR-free document understanding framework based on pretrained Multimodal Large Language Models (MLLMs). Our approach employs multi-scale visual features to effectively handle various font sizes within document images. To address the increasing costs of considering the multi-scale visual inputs for MLLMs, we propose the Hierarchical Visual Feature Aggregation (HVFA) module, designed to reduce the number of input tokens to LLMs. Leveraging a feature pyramid with cross-attentive pooling, our approach effectively manages the trade-off between information loss and efficiency without being affected by varying document image sizes. Furthermore, we introduce a novel instruction tuning task, which facilitates the model's text-reading capability by learning to predict the relative positions of input text, eventually minimizing the risk of truncated text caused by the limited capacity of LLMs. Comprehensive experiments validate the effectiveness of our approach, demonstrating superior performance in various document understanding tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。