让模型自由匹配文档中任意自定义区域,无需预设类别
Class-Agnostic Region-of-Interest Matching in Document Images
- 用双流网络+交叉注意力对齐参考与目标文档语义
- 在三个难度等级基准上表现良好,成为新任务基线
- 适合需要灵活提取文档信息的用户或系统开发者
文档理解与分析因广泛应用而备受关注。然而,现有方法如文档版面分析和关键信息提取仅适用于固定类别定义和粒度,难以实现用户自定义的灵活应用。为此,本文提出一项新任务——无类别区域匹配(Class-Agnostic Region-of-Interest Matching,简称RoI-Matching),旨在以灵活、高效、多粒度、开放集的方式匹配自定义区域。输入为参考文档与目标文档的视觉提示,输出为目标文档中的对应边界框。为满足需求,我们构建了基准数据集RoI-Matching-Bench,包含三个符合真实场景难度等级,并提出宏观与微观评估指标。此外,我们提出新框架RoI-Matcher,采用孪生网络在参考与目标域中提取多层级特征,并通过交叉注意力层融合与对齐跨域相似语义。实验表明,该方法虽流程简单,但在RoI-Matching-Bench上表现有效,可作为后续研究基线。代码已开源:https://github.com/pd162/RoI-Matching。
原文摘要 · Abstract (English)
Document understanding and analysis have received a lot of attention due to their widespread application. However, existing document analysis solutions, such as document layout analysis and key information extraction, are only suitable for fixed category definitions and granularities, and cannot achieve flexible applications customized by users. Therefore, this paper defines a new task named ``Class-Agnostic Region-of-Interest Matching'' (``RoI-Matching'' for short), which aims to match the customized regions in a flexible, efficient, multi-granularity, and open-set manner. The visual prompt of the reference document and target document images are fed into our model, while the output is the corresponding bounding boxes in the target document images. To meet the above requirements, we construct a benchmark RoI-Matching-Bench, which sets three levels of difficulties following real-world conditions, and propose the macro and micro metrics to evaluate. Furthermore, we also propose a new framework RoI-Matcher, which employs a siamese network to extract multi-level features both in the reference and target domains, and cross-attention layers to integrate and align similar semantics in different domains. Experiments show that our method with a simple procedure is effective on RoI-Matching-Bench, and serves as the baseline for further research. The code is available at https://github.com/pd162/RoI-Matching.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。