融合手写图像与轨迹数据，提升识别准确率
A Transformer Based Handwriting Recognition System Jointly Using Online and Offline Features
- 早期融合离线图像与在线轨迹特征
- 在IAMOn-DB上达到98.2%准确率，领先1%
- 适合需要高精度手写识别的场景
我们提出，手写识别可从栅格化字符图像和笔迹轨迹中获取互补信息，但现有系统多仅使用单一模态。本文设计端到端网络，在共享潜在空间中实现离线图像与在线笔迹数据的早期融合：图像通过补丁编码器转为固定长度视觉令牌，笔迹序列由轻量级Transformer编码为$(x, y, ext{pen})$嵌入。可学习的潜在查询同时关注两类流，生成上下文增强的笔画嵌入，经池化后解码并以交叉熵损失优化。因融合发生在高层分类前，时序线索在表征学习阶段相互强化，提升书写者无关性。在IAMOn-DB和VNOn-DB上的全面实验表明，该方法达到当前最优性能，最高超越此前最佳达1%。研究还展示了在ISI-Air数据集上通过手势化适配的可行性。代码已公开。
原文摘要 · Abstract (English)
We posit that handwriting recognition benefits from complementary cues carried by the rasterized complex glyph and the pen's trajectory, yet most systems exploit only one modality. We introduce an end-to-end network that performs early fusion of offline images and online stroke data within a shared latent space. A patch encoder converts the grayscale crop into fixed-length visual tokens, while a lightweight transformer embeds the $(x, y, \text{pen})$ sequence. Learnable latent queries attend jointly to both token streams, yielding context-enhanced stroke embeddings that are pooled and decoded under a cross-entropy loss objective. Because integration occurs before any high-level classification, temporal cues reinforce each other during representation learning, producing stronger writer independence. Comprehensive experiments on IAMOn-DB and VNOn-DB demonstrate that our approach achieves state-of-the-art accuracy, exceeding previous bests by up to 1\%. Our study also shows adaptation of this pipeline with gesturification on the ISI-Air dataset. Our code can be found here.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。