用事件相机识别场景文字,解决光照差、模糊等问题。
EventSTR: A Benchmark Dataset and Baselines for Event Stream based Scene Text Recognition
- 基于事件相机数据构建新识别框架,融合视觉编码与记忆增强。
- 在9928张高清事件图像上验证,中文英文字符识别准确率显著提升。
- 适合对低光、高速场景文本识别有需求的研究者和工业应用。
主流场景文字识别算法依赖于对光照敏感的RGB相机,难以应对低照度、运动模糊和杂乱背景等挑战。本文提出基于生物启发式事件相机的文字识别方法,构建了大规模基准数据集EventSTR,包含9,928张高分辨率(1280×720)事件样本,涵盖中英文字符。同时,我们为未来研究提供多个基线模型。此外,提出新型事件感知文字识别框架SimC-ESTR:先通过视觉编码器提取事件特征,再利用Q-former模块将其投影为视觉令牌;更重要的是,在输入大语言模型前,引入基于记忆机制的令牌增强策略;并在大语言模型内部嵌入基于相似性的错误纠正机制,利用上下文信息从根本上修正微小误判。在自建的EventSTR数据集及两个模拟STR数据集上的大量实验充分证明了所提模型的有效性。我们认为该数据集与模型创新性地提出了事件相机驱动的文字识别任务,有望推动事件相机在各行业的应用。源代码与预训练模型将公开于https://github.com/Event-AHU/EventSTR。
原文摘要 · Abstract (English)
Mainstream Scene Text Recognition (STR) algorithms are developed based on RGB cameras which are sensitive to challenging factors such as low illumination, motion blur, and cluttered backgrounds. In this paper, we propose to recognize the scene text using bio-inspired event cameras by collecting and annotating a large-scale benchmark dataset, termed EventSTR. It contains 9,928 high-definition (1280 * 720) event samples and involves both Chinese and English characters. We also benchmark multiple STR algorithms as the baselines for future works to compare. In addition, we propose a new event-based scene text recognition framework, termed SimC-ESTR. It first extracts the event features using a visual encoder and projects them into tokens using a Q-former module. More importantly, we propose to augment the vision tokens based on a memory mechanism before feeding into the large language models. A similarity-based error correction mechanism is embedded within the large language model to correct potential minor errors fundamentally based on contextual information. Extensive experiments on the newly proposed EventSTR dataset and two simulation STR datasets fully demonstrate the effectiveness of our proposed model. We believe that the dataset and algorithmic model can innovatively propose an event-based STR task and are expected to accelerate the application of event cameras in various industries. The source code and pre-trained models will be released on https://github.com/Event-AHU/EventSTR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。