测试视觉语言模型真实场景拼写纠错能力，发现表现远不如人类。
Vision Language Models Are Not (Yet) Spelling Correctors
- 构建首个跨中英文的真实图像拼写纠错基准ReViCo，支持图像与词粒度评估。
- 现有模型在纠错任务上显著落后于人类，尤其在复杂真实场景中。
- 提出联合识别与纠错、利用背景信息两种改进方案，有效提升性能。
视觉输入的拼写纠正对视觉语言模型（VLMs）构成独特挑战，因其需直接在图像中检测并修正文本错误。本文提出ReViCo（Real Visual Correction），首个系统评估VLMs在中英文真实世界图像拼写纠正上的基准。ReViCo包含从真实图像数据中收集的自然拼写错误，支持图像级与词粒度的细粒度评估。通过对代表性级联式（Qwen）与原生式（InternVL）开源模型，以及闭源系统（GPT-4o、Claude）的全面实验，我们发现当前VLMs在纠错能力上明显低于人类表现。为应对这些局限，我们探索两种解决方案：联合OCR-纠错管道与背景信息增强方法，二者均带来一致性能提升。分析揭示了现有架构的根本缺陷，并为推进多模态拼写纠正提供了可操作洞见。
原文摘要 · Abstract (English)
Spelling correction from visual input poses unique challenges for vision language models (VLMs), as it requires not only detecting but also correcting textual errors directly within images. We present ReViCo (Real Visual Correction), the first benchmark that systematically evaluates VLMs on real-world visual spelling correction across Chinese and English. ReViCo contains naturally occurring errors collected from real-world image data and supports fine-grained evaluation at both image and token levels. Through comprehensive experiments on representative cascaded (Qwen) and native (InternVL) open-source models, as well as closed-source systems (GPT-4o, Claude), we show that current VLMs fall significantly short of human performance, particularly in correction. To address these limitations, we explore two solution paradigms: a Joint OCR-Correction pipeline and a Background Information enhanced approach, both of which yield consistent performance gains. Our analysis highlights fundamental limitations of existing architectures and provides actionable insights for advancing multimodal spelling correction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。