arXiv:2507.09531cs.CVcs.AI2025-07

通过智能分块减少冗余计算,提升文档信息提取精度。

VDInstruct: Zero-Shot Key Information Extraction via Content-Aware Vision Tokenization

  • 按文档复杂度动态生成视觉令牌,避免均匀切分
  • 在多个基准上达到最优,令牌数减少3.6倍
  • 零样本表现优于主流模型,适合未见过的文档

关键信息提取(KIE)对理解视觉文档(如收据和合同)至关重要,需精确提取语义内容并准确捕捉空间结构。现有多模态大模型在密集文档上表现不佳,且依赖随图像尺寸增长的视觉令牌化方法,导致计算冗余与内存效率低下。为此,我们提出VDInstruct,一种将空间区域检测与语义特征提取分离的MLLM。其核心是内容感知的令牌化策略:不均匀切分图像,而是按文档复杂度生成令牌,保留关键结构并消除冗余。通过三阶段训练,该模型在KIE基准上达到当前最优(SOTA)性能,精度匹配或超越领先方法,同时图像令牌数量减少约3.6倍。零样本评估中,VDInstruct比DocOwl 1.5等强基线高出+5.5 F1点,展现出对未见文档的强鲁棒性。结果表明,结合内容感知令牌化与显式版式建模,是文档理解的可行新方向。数据、源码与模型权重将公开。

原文摘要 · Abstract (English)

Key Information Extraction (KIE) underpins the understanding of visual documents (e.g., receipts and contracts) by extracting precise semantic content and accurately capturing spatial structure. Yet existing multimodal large language models (MLLMs) often perform poorly on dense documents and rely on vision tokenization approaches that scale with image size, leading to redundant computation and memory inefficiency. To address these challenges, we introduce VDInstruct, an MLLM that separates spatial region detection from semantic feature extraction. Central to our model is a content-aware tokenization strategy: rather than fragmenting the entire image uniformly, it generates tokens in proportion to document complexity, preserving critical structure while eliminating wasted tokens. Leveraging a three-stage training paradigm, our model achieves state-of-the-art (SOTA) results on KIE benchmarks, matching or exceeding the accuracy of leading approaches while reducing the number of image tokens by roughly 3.6x. In zero-shot evaluations, VDInstruct surpasses strong baselines-such as DocOwl 1.5-by +5.5 F1 points, highlighting its robustness to unseen documents. These findings show that content-aware tokenization combined with explicit layout modeling offers a promising direction forward for document understanding. Data, source code, and model weights will be made publicly available.

文档理解视觉令牌化KIE零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。