构建统一基准评估大模型从文档中提取关键信息的能力
UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents
- 设计双轨评测框架，覆盖预定义与开放类别信息抽取
- 15个顶尖大模型在复杂版式下表现明显下降，长尾字段识别率低
- 适合关注文档智能、多模态模型评估的研究者和开发者
真实世界文档中的关键信息抽取（KIE）因版式差异大、视觉质量参差及任务需求多样而极具挑战。近年来的大规模多模态模型（LMMs）展现出直接从文档图像端到端完成KIE的潜力。为系统评估不同实际应用场景下的性能表现，我们提出UNIKIE-BENCH，一个统一的基准测试平台。该基准包含两个互补赛道：受限类别赛道采用场景预设的结构化模式，贴近实际应用需求；开放类别赛道则提取文档中明确存在的任意关键信息。对15个主流LMMs的实验表明，模型在不同模式定义、长尾关键字段及复杂布局下均出现显著性能下降，且在不同文档类型和场景间存在明显性能差异。这些结果凸显了当前基于LMM的KIE在定位准确性和版式感知推理方面的持续挑战。所有代码与数据集已公开于https://github.com/NEUIR/UNIKIE-BENCH。
原文摘要 · Abstract (English)
Key Information Extraction (KIE) from real-world documents remains challenging due to substantial variations in layout structures, visual quality, and task-specific information requirements. Recent Large Multimodal Models (LMMs) have shown promising potential for performing end-to-end KIE directly from document images. To enable a comprehensive and systematic evaluation across realistic and diverse application scenarios, we introduce UNIKIE-BENCH, a unified benchmark designed to rigorously evaluate the KIE capabilities of LMMs. UNIKIE-BENCH consists of two complementary tracks: a constrained-category KIE track with scenario-predefined schemas that reflect practical application needs, and an open-category KIE track that extracts any key information that is explicitly present in the document. Experiments on 15 state-of-the-art LMMs reveal substantial performance degradation under diverse schema definitions, long-tail key fields, and complex layouts, along with pronounced performance disparities across different document types and scenarios. These findings underscore persistent challenges in grounding accuracy and layout-aware reasoning for LMM-based KIE. All codes and datasets are available at https://github.com/NEUIR/UNIKIE-BENCH.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。