专为文档与网页智能体设计的视觉编码器,提升结构化信息理解能力。
DAVE: A VLM Vision Encoder for Document Understanding and Web Agents
- 通过自监督+自回归训练,用无标注数据学习文档与网页图像特征
- 在文档解析与网页定位任务上优于现有模型,准确率显著提升
- 适配多种网页智能体架构,适合需要结构化视觉理解的场景
尽管视觉语言模型在多模态任务中表现优异,但其视觉编码器缺乏对文档理解与网页智能体至关重要的低级结构与空间信息。为此,我们提出DAVE,一种专为视觉语言模型设计、面向文档与网页智能体任务的视觉编码器。训练流程利用大量无标注数据,避免昂贵的标注成本:先在无标注图像上进行自监督预训练,再在少量高质量数据上进行有监督自回归预训练,学习解析与定位等任务。在有监督阶段,采用两种策略增强编码器与各类任务的对齐性:(i) 提出新型模型融合方法,结合使用不同文本解码器训练的编码器,确保与多种网页智能体架构的兼容性;(ii) 通过集成训练,融合通用型编码器(如SigLIP2)与专用于文档和网页的表征。在经典文档任务、视觉问答、网页定位及基于代理的基准测试中,实验验证了该方法的有效性,确立了DAVE作为文档与网页应用中强健视觉编码器的地位。
原文摘要 · Abstract (English)
While Vision-language models (VLMs) have demonstrated remarkable performance across multi-modal tasks, their choice of vision encoders presents a fundamental weakness: their low-level features lack the robust structural and spatial information essential for document understanding and web agents. To bridge this gap, we introduce DAVE, a vision encoder purpose-built for VLMs and tailored for these tasks. Our training pipeline is designed to leverage abundant unlabeled data to bypass the need for costly large-scale annotations for document and web images. We begin with a self-supervised pretraining stage on unlabeled images, followed by a supervised autoregressive pretraining stage, where the model learns tasks like parsing and localization from limited, high-quality data. Within the supervised stage, we adopt two strategies to improve our encoder's alignment with both general visual knowledge and diverse document and web agentic tasks: (i) We introduce a novel model-merging scheme, combining encoders trained with different text decoders to ensure broad compatibility with different web agentic architectures. (ii) We use ensemble training to fuse features from pretrained generalist encoders (e.g., SigLIP2) with our own document and web-specific representations. Extensive experiments on classic document tasks, VQAs, web localization, and agent-based benchmarks validate the effectiveness of our approach, establishing DAVE as a strong vision encoder for document and web applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。