让视觉与文本信息互补，提升表格理解能力
How to Utilize Complementary Vision-Text Information for 2D Structure Understanding
- 用视觉信息动态查询文本，提炼关键内容
- 在13个表格数据集上比纯文本基线提升23.9%
- 适合需要精准表格理解的场景，如文档分析
大语言模型通常将二维表格线性化为一维序列以适应自回归架构，这削弱了行-列邻接关系及其他布局线索。而纯视觉编码器虽能捕捉空间信息，却常难以保留单元格内的精确文本。我们的分析表明，这两种模态为大语言模型提供了高度不同的信息，具有强互补性。然而，直接拼接等融合方法效果有限，且常引入跨模态干扰。为此，我们提出DiVA-Former，一种轻量级架构，可有效整合视觉与文本信息。DiVA-Former利用视觉标记作为动态查询，将长文本序列压缩为紧凑向量，从而充分利用视觉-文本的互补信息。在13个表格基准测试上，DiVA-Former相比纯文本基线提升23.9%，并持续优于使用视觉、文本或两者结合的现有基线。
原文摘要 · Abstract (English)
LLMs typically linearize 2D tables into 1D sequences to fit their autoregressive architecture, which weakens row-column adjacency and other layout cues. In contrast, purely visual encoders can capture spatial cues, yet often struggle to preserve exact cell text. Our analysis reveals that these two modalities provide highly distinct information to LLMs and exhibit strong complementarity. However, direct concatenation and other fusion methods yield limited gains and frequently introduce cross-modal interference. To address this issue, we propose DiVA-Former, a lightweight architecture designed to effectively integrate vision and text information. DiVA-Former leverages visual tokens as dynamic queries to distill long textual sequences into digest vectors, thereby effectively exploiting complementary vision--text information. Evaluated across 13 table benchmarks, DiVA-Former improves upon the pure-text baseline by 23.9\% and achieves consistent gains over existing baselines using visual inputs, textual inputs, or a combination of both.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。