用分层点建模实现端到端视觉信息提取，兼顾准确与可解释性。
HIP: Hierarchical Point Modeling and Pre-training for Visual Information Extraction
- 将实体建模为分层点，统一文本定位、分组与分类任务
- 在多个公开数据集上超越现有最优方法，准确率显著提升
- 适合需要高可解释性的文档理解场景
端到端视觉信息提取（VIE）旨在将文本定位、单词分组和实体标注等层次化子任务整合到统一框架中。处理各子任务间的差异是设计高效VIE模型的关键。依赖OCR的方法严重依赖离线OCR引擎，不可避免地受制于识别错误；而无OCR方法，尤其是采用黑箱模型的，可能产生不可解释或幻觉内容。受CenterNet、DeepSolo和ESP启发，本文提出HIP，将实体建模为分层点，更贴合端到端VIE任务的层级结构。这些分层点可灵活编码，并解码为期望的文本转录、各类区域中心及实体类别。此外，设计了图像重建、版面学习和语言增强三类分层预训练策略，以强化跨模态表示能力。定量实验表明，HIP在多个公开基准上优于当前最优方法，定性结果亦显示其出色可解释性。
原文摘要 · Abstract (English)
End-to-end visual information extraction (VIE) aims at integrating the hierarchical subtasks of VIE, including text spotting, word grouping, and entity labeling, into a unified framework. Dealing with the gaps among the three subtasks plays a pivotal role in designing an effective VIE model. OCR-dependent methods heavily rely on offline OCR engines and inevitably suffer from OCR errors, while OCR-free methods, particularly those employing a black-box model, might produce outputs that lack interpretability or contain hallucinated content. Inspired by CenterNet, DeepSolo, and ESP, we propose HIP, which models entities as HIerarchical Points to better conform to the hierarchical nature of the end-to-end VIE task. Specifically, such hierarchical points can be flexibly encoded and subsequently decoded into desired text transcripts, centers of various regions, and categories of entities. Furthermore, we devise corresponding hierarchical pre-training strategies, categorized as image reconstruction, layout learning, and language enhancement, to reinforce the cross-modality representation of the hierarchical encoders. Quantitative experiments on public benchmarks demonstrate that HIP outperforms previous state-of-the-art methods, while qualitative results show its excellent interpretability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。