arXiv:2502.06100cs.CVeess.SP2025-02被引 3

提出单流多模态手写文本识别框架,兼顾精度与效率

Col-OLHTR: A Novel Framework for Multimodal Online Handwritten Text Recognition

  • 通过协同学习融合轨迹与图像特征,训练时用双流监督,推理时仅用单流
  • 在多个基准上达到当前最优性能,字符错误率显著降低
  • 适合需要高效推理的实时手写识别场景

在线手写文本识别(OLHTR)因其广泛应用而受到广泛关注。现有方法通常将任务视为序列识别,使用单一轨迹或图像编码器,或采用多流编码器结合CTC或注意力解码器。但这些方法存在不足:1)单编码器往往只关注局部轨迹或视觉区域,难以动态捕捉复杂情况下的全局特征;2)多流编码器虽更全面,却结构复杂、推理成本高。为此,我们提出一种基于协同学习的OLHTR框架——Col-OLHTR,该框架在训练阶段学习多模态特征,推理阶段保持单流结构。Col-OLHTR包含轨迹编码器、点到空间对齐(P2SA)模块和注意力解码器。P2SA模块利用轨迹特征与二维旋转位置编码学习图像级空间特征。训练时,额外引入图像流编码器-解码器,为P2SA特征提供监督信号;推理时丢弃额外流,仅保留P2SA模块并融合后输入解码器,简化流程同时保持高性能。在多个OLHTR基准上的实验结果表明,该方法达到当前最优(SOTA)性能,验证了设计的有效性与鲁棒性。

原文摘要 · Abstract (English)

Online Handwritten Text Recognition (OLHTR) has gained considerable attention for its diverse range of applications. Current approaches usually treat OLHTR as a sequence recognition task, employing either a single trajectory or image encoder, or multi-stream encoders, combined with a CTC or attention-based recognition decoder. However, these approaches face several drawbacks: 1) single encoders typically focus on either local trajectories or visual regions, lacking the ability to dynamically capture relevant global features in challenging cases; 2) multi-stream encoders, while more comprehensive, suffer from complex structures and increased inference costs. To tackle this, we propose a Collaborative learning-based OLHTR framework, called Col-OLHTR, that learns multimodal features during training while maintaining a single-stream inference process. Col-OLHTR consists of a trajectory encoder, a Point-to-Spatial Alignment (P2SA) module, and an attention-based decoder. The P2SA module is designed to learn image-level spatial features through trajectory-encoded features and 2D rotary position embeddings. During training, an additional image-stream encoder-decoder is collaboratively trained to provide supervision for P2SA features. At inference, the extra streams are discarded, and only the P2SA module is used and merged before the decoder, simplifying the process while preserving high performance. Extensive experimental results on several OLHTR benchmarks demonstrate the state-of-the-art (SOTA) performance, proving the effectiveness and robustness of our design.

手写识别多模态单流推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。