测试十种OCR模型在印地语文字上的表现，发现英文强模型不适用于印地语。
Can OCR-VLMs Read Devanagari? A Stress-Test Benchmark and Post-Correction Study
- 构建合成与真实印刷文本的应力测试基准，评估多种OCR模型表现。
- 专用OCR-VLM在退化条件下易出现灾难性重复错误，平均性能被严重拉低。
- 公开数据集与代码，适合研究多语言、低资源文本识别的学者使用。
从传统引擎到专用OCR视觉语言模型（OCR-VLMs）及前沿多模态大模型，现有系统在英文和中文文档上表现优异，但在印度语系文字（如德纳加里文/印地语）上的表现尚不明确。本研究在四种合成退化条件和300个真实印刷扫描件上，对十种系统进行了基准测试：经典EasyOCR；开源VLM（Qwen2.5-VL-3B、Qwen3-VL-8B、olmOCR-7B）；专用OCR-VLM（DeepSeek-OCR、Unlimited-OCR）；以及前沿闭源模型（Gemini 2.5 Flash、Claude Opus 4.7、GPT-5.5、Mistral OCR）。结果表明：在干净渲染文本中，所有系统均达到chrF++ 91至98，合成数据无法有效区分；在退化情况下，专用OCR-VLM最脆弱，DeepSeek-OCR出现罕见但灾难性的重复错误（输出长度达参考值的71倍），导致整体均值崩溃，因此采用中位数与灾难率作为核心指标；在真实扫描件上，九种系统性能大幅下滑（EasyOCR从93.6降至58.3），性能范围跨度达76分，说明合成数据严重高估了实际表现；强英文识别能力无法迁移至印地语：GPT-5.5下降至58.5（与EasyOCR持平），olmOCR-7B降至40.5，而开源的Qwen3-VL-8B（75.2，单卡24GB可运行）优于GPT-5.5并接近Mistral；Gemini与Claude领先，分别为86.3与82.2。通过字节级后校正（ByT5）分析，表面错误（数字、标点）与结构错误（连写、元音符号、鼻音点）可区分，低成本模型自身校正提升1.2–1.5分，但跨模型泛化失败。研究已发布基准、代码与模型。
原文摘要 · Abstract (English)
OCR systems, ranging from classical engines to specialised OCR vision-language models (OCR-VLMs) and frontier multimodal LLMs, report strong results on English and Chinese document benchmarks, yet their behaviour on Indic scripts is largely uncharacterised. We benchmark ten systems on Devanagari (Hindi): classical EasyOCR; open VLMs (Qwen2.5-VL-3B, Qwen3-VL-8B, olmOCR-7B); specialised OCR-VLMs (DeepSeek-OCR, Unlimited-OCR); and frontier closed models (Gemini 2.5 Flash, Claude Opus 4.7, GPT-5.5, Mistral OCR), across four synthetic degradation conditions and 300 real printed scans. We report four findings. First, on clean rendered text all ten cluster within chrF++ 91 to 98, so synthetic text does not separate them. Second, under degradation the specialised OCR-VLMs are the most fragile: DeepSeek-OCR suffers rare but catastrophic repetition failures (outputs up to 71 the reference length) that wreck its corpus mean even though its median is the best of any system, which is why we report median and catastrophic-rate instead of the mean. Third, on real scans nine of the ten systems collapse (EasyOCR falls from chrF++ 93.6 to 58.3) and the field spreads across a 76-point range, so synthetic renders badly overstate Devanagari quality. Fourth, strong English OCR does not predict Indic OCR: GPT-5.5 drops to chrF++ 58.5 (tying classical EasyOCR) and olmOCR-7B, the model behind olmOCR-Bench, falls to 40.5, while the open Qwen3-VL-8B (75.2, runnable on a single 24 GB GPU) beats GPT-5.5 and approaches Mistral; Gemini and Claude lead at 86.3 and 82.2. An error taxonomy separates surface errors (numerals, punctuation) from structural ones (conjuncts, matras, nukta), and a byte-level (ByT5) post-corrector improves a cheap engine on its own error distribution (chrF++ +1.2 to +1.5) but does not transfer across engines. We release the benchmark, code, and models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。