让文档中的图文信息统一转为可解析的文本,提升文档理解能力。
Multimodal OCR: Parse Anything from Documents
- 将文本与图表等视觉元素一同解析为结构化输出,保留语义关系。
- 在olmOCR Bench上达到83.9分,仅次于Gemini 3 Pro,超越开源系统。
- 适合需要高精度图文联合解析的研究与应用,如文档自动化处理。
我们提出多模态OCR(MOCR),一种将文档中的文字与图形统一解析为结构化文本表示的新范式。不同于传统OCR仅识别文字、丢弃图形区域,我们的方法dots.mocr将图表、流程图、表格、图标等视觉元素作为核心解析目标,支持端到端训练,利用图文之间的语义关联。该方法可重建文字与图形内容,实现更真实的文档还原;并把原本被忽略的图形转化为代码级监督信号,释放现有文档中的多模态监督潜力。为实现大规模应用,我们构建了涵盖PDF、渲染网页和原生SVG资产的数据引擎,并通过分阶段预训练与监督微调,训练了一个30亿参数的紧凑模型。在文档解析与结构化图形解析双重评估中,该模型在OCR Arena Elo排行榜中位列第二,优于现有开源系统,在olmOCR Bench上取得83.9的新纪录;在图像转SVG任务中,对图表、界面布局、科学图示及化学结构图的重建质量均超过Gemini 3 Pro。结果表明,该方法为构建大规模图文代码语料库提供了可扩展路径。代码与模型已公开于https://github.com/rednote-hilab/dots.mocr。
原文摘要 · Abstract (English)
We present Multimodal OCR (MOCR), a document parsing paradigm that jointly parses text and graphics into unified textual representations. Unlike conventional OCR systems that focus on text recognition and leave graphical regions as cropped pixels, our method, termed dots.mocr, treats visual elements such as charts, diagrams, tables, and icons as first-class parsing targets, enabling systems to parse documents while preserving semantic relationships across elements. It offers several advantages: (1) it reconstructs both text and graphics as structured outputs, enabling more faithful document reconstruction; (2) it supports end-to-end training over heterogeneous document elements, allowing models to exploit semantic relations between textual and visual components; and (3) it converts previously discarded graphics into reusable code-level supervision, unlocking multimodal supervision embedded in existing documents. To make this paradigm practical at scale, we build a comprehensive data engine from PDFs, rendered webpages, and native SVG assets, and train a compact 3B-parameter model through staged pretraining and supervised fine-tuning. We evaluate dots.mocr from two perspectives: document parsing and structured graphics parsing. On document parsing benchmarks, it ranks second only to Gemini 3 Pro on our OCR Arena Elo leaderboard, surpasses existing open-source document parsing systems, and sets a new state of the art of 83.9 on olmOCR Bench. On structured graphics parsing, our model achieves higher reconstruction quality than Gemini 3 Pro across image-to-SVG benchmarks, demonstrating strong performance on charts, UI layouts, scientific figures, and chemical diagrams. These results show a scalable path toward building large-scale image-to-code corpora for multimodal pretraining. Code and models are publicly available at https://github.com/rednote-hilab/dots.mocr.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。