结构化输入让多模态模型更懂文档，效果远超原始文字
Structured Attention Matters to Multimodal LLMs in Document Understanding
- 用LaTeX格式保留文档层级与空间关系，避免注意力分散
- 在多种文档上提升问答准确率，无需改模型或额外训练
- 适合需要精准理解复杂文档的场景，如法律、医疗
文档理解仍是多模态大模型（MLLMs）的重大挑战。现有研究多聚焦于通过精确的多模态查询定位证据页，而本文首次揭示了一个被忽视的关键问题：输入格式如何影响文档理解性能。系统分析发现，原始OCR文本常会降低而非提升MLLMs表现，这一反直觉现象归因于注意力分散和结构信息丢失。为此，我们提出一种新颖的结构保持方法，采用LaTeX范式编码文档元素，保留关键的层级组织与空间关系。注意力分析显示，结构化文本能诱导出对文本和视觉内容均具结构性的注意力模式，使模型聚焦于语义有意义区域，减少注意力浪费。该方法显著提升了多种文档类型下的问答性能，且无需架构修改或额外训练。
原文摘要 · Abstract (English)
Document understanding remains a significant challenge for multimodal large language models (MLLMs). While previous research has primarily focused on locating evidence pages through precise multimodal queries, our work investigates a fundamental yet overlooked aspect: how input format influences document comprehension performance. Through systematic analysis, we discover that raw OCR text often impairs rather than improves MLLMs' performance, which is a counterintuitive finding we attribute to attention dispersion and structure loss. To further substantiate our hypothesis, we propose a novel structure-preserving approach that encodes document elements using the LaTex paradigm, maintaining the hierarchical organization and spatial relationships critical for comprehension. Our attention analysis reveals that structured text induces structured attention patterns on both textual and visual content, directing models to focus on semantically meaningful regions while reducing attention waste. This approach significantly enhances MLLMs' document question answering performance across diverse document types without requiring architectural modifications or additional training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。