构建多模态文档结构化提取基准,揭示小模型关键瓶颈
VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents
- 通过逆向标注生成带确定性真值的合成政府表单数据
- 40亿参数以下模型主要受限于结构合规性而非提取能力
- 布局保留文本比图像更有效,20亿参数微调可提升81个百分点
我们提出VAREX(VARied-schema EXtraction),一个用于评估多模态基础模型从政府表单中提取结构化数据的基准。VAREX采用逆向标注流程,程序化地向PDF模板填充合成数据,生成可通过三阶段质量保证验证的确定性真值。该基准包含1,777份文档,覆盖1,771个独特数据模式,分为三类结构,每份文档提供四种输入模态:纯文本、保留版式文本(对齐空白以近似列位置)、文档图像,或文本与图像联合。不同于以往仅使用单一输入形式的基准,VAREX为每份文档提供四种受控模态,可系统分析输入格式对提取准确率的影响。我们评估了20个模型,涵盖前沿商业模型至小型开源模型,特别关注≤4B参数、适用于成本敏感和延迟受限部署的模型。结果表明:(1) 参数低于4B时,结构输出合规性(非提取能力)是主要瓶颈;尤其在出现‘模式回声’现象(模型生成符合模式结构但无实际值)的模型中,得分下降45-65个百分点;(2) 在2B参数下进行抽取专项微调,准确率提升+81个百分点,证明指令遵循缺陷无需依赖规模即可解决;(3) 布局保留文本带来最大增益(+3-18个百分点),优于像素级视觉线索;(4) 该基准在60%-95%准确率区间最具区分度。数据集与评估代码已公开。
原文摘要 · Abstract (English)
We introduce VAREX (VARied-schema EXtraction), a benchmark for evaluating multimodal foundation models on structured data extraction from government forms. VAREX employs a Reverse Annotation pipeline that programmatically fills PDF templates with synthetic values, producing deterministic ground truth validated through three-phase quality assurance. The benchmark comprises 1,777 documents with 1,771 unique schemas across three structural categories, each provided in four input modalities: plain text, layout-preserving text (whitespace-aligned to approximate column positions), document image, or both text and image combined. Unlike existing benchmarks that evaluate from a single input representation, VAREX provides four controlled modalities per document, enabling systematic ablation of how input format affects extraction accuracy -- a capability absent from prior benchmarks. We evaluate 20 models from frontier proprietary models to small open models, with particular attention to models <=4B parameters suitable for cost-sensitive and latency-constrained deployment. Results reveal that (1) below 4B parameters, structured output compliance -- not extraction capability -- is a dominant bottleneck; in particular, schema echo (models producing schema-conforming structure instead of extracted values) depresses scores by 45-65 pp (percentage points) in affected models; (2) extraction-specific fine-tuning at 2B yields +81 pp gains, demonstrating that the instruction-following deficit is addressable without scale; (3) layout-preserving text provides the largest accuracy gain (+3-18 pp), exceeding pixel-level visual cues; and (4) the benchmark most effectively discriminates models in the 60-95% accuracy band. Dataset and evaluation code are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。