用多模态方法精准连接历史地图上的分散文字,提升文本理解能力。
LIGHT: Multi-Modal Text Linking on Historical Maps
- 融合语言、图像和几何信息,建模文字位置与形状关系。
- 在ICDAR 2024/2025竞赛数据上超越现有方法,读序预测更准确。
- 适合历史地理、数字人文研究者,尤其关注地图文本解析的场景。
历史地图上的文本为历史、经济、地理等研究提供宝贵信息。与结构化或半结构化文档不同,地图文本在方向、阅读顺序、形状和布局上差异显著。尽管现代方法可检测并转录文本区域,但在有效关联识别出的文本片段(如多词地名)方面仍面临挑战。现有布局分析方法依赖语言特征建模词间关系,却忽视对地图文本至关重要的几何信息。为此,我们提出LIGHT——一种融合语言、图像与几何特征的多模态文本链接方法。LIGHT引入几何感知嵌入模块,编码文本区域的多边形坐标,捕捉形状及相对空间位置;并将该几何信息与LayoutLMv3生成的视觉与语言标记嵌入统一融合。通过跨模态信息,采用双向学习策略直接预测每个文本实例的阅读顺序后继项,增强序列鲁棒性。实验表明,LIGHT在ICDAR 2024/2025 MapText竞赛数据集上优于现有方法,验证了多模态学习在历史地图文本链接中的有效性。
原文摘要 · Abstract (English)
Text on historical maps provides valuable information for studies in history, economics, geography, and other related fields. Unlike structured or semi-structured documents, text on maps varies significantly in orientation, reading order, shape, and placement. Many modern methods can detect and transcribe text regions, but they struggle to effectively ``link'' the recognized text fragments, e.g., determining a multi-word place name. Existing layout analysis methods model word relationships to improve text understanding in structured documents, but they primarily rely on linguistic features and neglect geometric information, which is essential for handling map text. To address these challenges, we propose LIGHT, a novel multi-modal approach that integrates linguistic, image, and geometric features for linking text on historical maps. In particular, LIGHT includes a geometry-aware embedding module that encodes the polygonal coordinates of text regions to capture polygon shapes and their relative spatial positions on an image. LIGHT unifies this geometric information with the visual and linguistic token embeddings from LayoutLMv3, a pretrained layout analysis model. LIGHT uses the cross-modal information to predict the reading-order successor of each text instance directly with a bi-directional learning strategy that enhances sequence robustness. Experimental results show that LIGHT outperforms existing methods on the ICDAR 2024/2025 MapText Competition data, demonstrating the effectiveness of multi-modal learning for historical map text linking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。