用两阶段框架自动数字化多语种词典，解决扫描版难读难题。
MUDIDI: A Two-Stage Framework for Multilingual Dictionary Digitization with Language Models
- 分两步处理：先验评估字符与排版识别质量，再分割词条并转为标准格式。
- 大语言模型在多数语系中表现优于传统OCR和视觉语言模型。
- 适合从事濒危语言保护、数字人文的学者与技术团队使用。
多语种词典是低资源及濒危语言最宝贵的文献资料，但许多仍仅以扫描件形式存在。长期以来，因语言特有文字系统、复杂的双栏布局及缩写、交叉引用等，将其数字化为机器可读格式几乎不可能。近期视觉-语言模型提供了新可能，但其对字符、标记及词典结构的保留效果尚不明确。本文提出MUDIDI，一种两阶段多语种词典数字化框架：第一阶段评估字符识别与标记保真度；第二阶段聚焦词条分割，并映射至SIL的Multi-Dictionary Formatter标准结构。同时发布一个包含30个公开词典的人工标注数据集，涵盖多样书写系统、语系与格式。我们在该数据集上基准测试了OCR系统、通用大语言模型（LLMs）与视觉语言模型（VLMs），结果表明LLMs在多数语系与语言中两阶段表现更优，并提供针对挑战性场景的实用优化建议。最后发现，向模型补充词典前言等额外信息可进一步提升数字化质量。
原文摘要 · Abstract (English)
Multilingual dictionaries are among the most valuable documentary resources for low-resource and endangered languages, yet many remain available only as scans. For many decades, their digitization and conversion into a machine-readable format was nearly impossible due to language-specific scripts, complex multi-column layouts full of entries with abbreviations and cross-references. Recent vision-language models offer a promising solution, but it is unclear how well they preserve characters, markup, and process lexicographic structure. We introduce MUDIDI, a two-stage framework for multi-lingual dictionary digitization. Stage One evaluates the quality of character recognition and markup preservation; Stage Two focuses on dictionary entry segmentation with subsequent mapping into a machine-readable lexicographic schema, SIL's Multi-Dictionary Formatter. We also release a dataset that consists of human-annotated lexicographic entries collected from 30 public-domain dictionaries featuring diverse writing systems, language families, and formats. We benchmark OCR systems, general-purpose Large Language Models (LLMs), and Vision Language Models (VLMs) on the dataset, demonstrating superior performance of LLMs across most writing systems and languages in both stages, and provide practical guidelines on improving the results for more challenging scenarios. Finally, we show that supplementing additional information, such as dictionary introduction, to the LLMs can improve the quality of the digitized dictionary. Github: https://github.com/DavidSamuell/MUDIDI-Pipeline-for-Digitization-of-Multilingual-Dictionary/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。