arXiv:2504.02971cs.CVcs.CL2025-04CVPR

让视觉模型精准聚焦查询区域,小数据下表现更优。

QID: Efficient Query-Informed ViTs in Data-Scarce Regimes for OCR-free Visual Document Understanding

  • 将查询嵌入注入视觉编码器,不改变原有结构
  • 在标注少的数据上显著提升识别准确率
  • 适合文档理解中缺少文字标注的场景

在视觉文档理解任务中,用新数据微调预训练视觉-语言模型时,往往难以优化视觉编码器以识别文本密集图像中的查询相关区域。现有方法通过修改网络结构直接注入查询,但在标注有限的新数据集上适应性差。为此,我们提出QID,一种轻量级、保持架构不变的方法,将查询嵌入融入视觉编码器,在数据稀缺的微调场景下取得显著性能提升。具体地,该方法采用双模块框架:查询感知模块生成专属查询向量,精准引导模型关注;查询无关模块捕捉标记间的空间关系,保障鲁棒的空间理解。两个模块均独立于视觉注意力块运行,实现查询嵌入的针对性学习,增强视觉语义识别能力。在多个数据集上的实验表明,使用OCR-free VLMs时,本方法在文本密集文档处理中尤其有效,尤其在数据稀缺环境下表现突出。

原文摘要 · Abstract (English)

In Visual Document Understanding (VDU) tasks, fine-tuning a pre-trained Vision-Language Model (VLM) with new datasets often falls short in optimizing the vision encoder to identify query-specific regions in text-rich document images. Existing methods that directly inject queries into model layers by modifying the network architecture often struggle to adapt to new datasets with limited annotations. To address this, we introduce QID, a novel, streamlined, architecture-preserving approach that integrates query embeddings into the vision encoder, leading to notable performance gains, particularly in data-scarce fine-tuning scenarios. Specifically, our approach introduces a dual-module framework: a query-aware module that generates a unique query vector to precisely guide the model's focus, as well as a query-agnostic module that captures the positional relationships among tokens, ensuring robust spatial understanding. Notably, both modules operate independently of the vision attention blocks, facilitating targeted learning of query embeddings and enhancing visual semantic identification. Experiments with OCR-free VLMs across multiple datasets demonstrate significant performance improvements using our method, especially in handling text-rich documents in data-scarce environments.

视觉文档理解小样本学习查询引导ViT

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。