用可变形卷积与检索增强生成，提升图像文字识别准确率
DOTA: Deformable Optimized Transformer Architecture for End-to-End Text Recognition with Retrieval-Augmented Generation
- 结合可变形卷积和检索增强生成，优化文字特征提取
- 在6个数据集上平均准确率达77.77%，最高达97.32%
- 适合需要高精度端到端文字识别的场景
自然图像中的文字识别仍是计算机视觉与自然语言处理中重要且具挑战性的任务。本文提出一种新型端到端框架，融合ResNet与Vision Transformer主干网络，引入可变形卷积、检索增强生成（Retrieval-Augmented Generation）及条件随机场（CRF）。具体而言，在第三、四阶段块中以可变形卷积替代标准卷积，采用自适应丢弃进行正则化，并利用CRF实现更精细的序列建模。在六个基准数据集IC13、IC15、SVT、IIIT5K、SVTP和CUTE80上的大量实验表明，该方法显著提升性能：分别取得97.32%（IC13）、58.26%（IC15）、88.10%（SVT）、74.13%（IIIT5K）、82.17%（SVTP）和66.67%（CUTE80）的准确率，平均准确率为77.77%，刷新文本识别新纪录，验证了其在多样化挑战性数据集上的鲁棒性。
原文摘要 · Abstract (English)
Text recognition in natural images remains a challenging yet essential task, with broad applications spanning computer vision and natural language processing. This paper introduces a novel end-to-end framework that combines ResNet and Vision Transformer backbones with advanced methodologies, including Deformable Convolutions, Retrieval-Augmented Generation, and Conditional Random Fields (CRF). These innovations collectively enhance feature representation and improve Optical Character Recognition (OCR) performance. Specifically, the framework substitutes standard convolution layers in the third and fourth blocks with Deformable Convolutions, leverages adaptive dropout for regularization, and incorporates CRF for more refined sequence modeling. Extensive experiments conducted on six benchmark datasets IC13, IC15, SVT, IIIT5K, SVTP, and CUTE80 validate the proposed method's efficacy, achieving notable accuracies: 97.32% on IC13, 58.26% on IC15, 88.10% on SVT, 74.13% on IIIT5K, 82.17% on SVTP, and 66.67% on CUTE80, resulting in an average accuracy of 77.77%. These results establish a new state-of-the-art for text recognition, demonstrating the robustness of the approach across diverse and challenging datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。