arXiv:2508.06107cs.CVcs.AI2025-08被引 1

用自监督注意力学习手写公式,无需标注数据也能精准识别。

Mask & Match: Learning to Recognize Handwritten Math with Self-Supervised Attention

  • 通过渐进式掩码训练注意力机制,自动聚焦运算符和嵌套结构。
  • 在CROHME数据集上达到新纪录,优于有监督与现有自监督方法。
  • 适合缺乏标注数据的数学公式识别场景,尤其手写输入应用。

手写数学表达式识别因二维结构、符号尺度不一及复杂空间关系而极具挑战。本文提出一种自监督学习框架,无需昂贵标注数据即可实现高效识别。方法首先结合全局与局部对比损失预训练图像编码器,以学习整体与细粒度表征;核心创新为一种新型自监督注意力网络,采用渐进式空间掩码策略训练,可无监督地学习运算符、指数、嵌套符号等语义关注区域。该掩码课程设计使模型对缺失或遮挡信息更具鲁棒性,提升结构理解能力。完整流程包含:(1)编码器自监督预训练,(2)注意力机制自监督学习,(3)使用Transformer解码器进行有监督微调生成LATEX序列。在CROHME基准上的大量实验表明,本方法超越现有自监督与全监督基线,验证了渐进注意力机制的有效性。

原文摘要 · Abstract (English)

Recognizing handwritten mathematical expressions (HMER) is a challenging task due to the inherent two-dimensional structure, varying symbol scales, and complex spatial relationships among symbols. In this paper, we present a self-supervised learning (SSL) framework for HMER that eliminates the need for expensive labeled data. Our approach begins by pretraining an image encoder using a combination of global and local contrastive loss, enabling the model to learn both holistic and fine-grained representations. A key contribution of this work is a novel self-supervised attention network, which is trained using a progressive spatial masking strategy. This attention mechanism is designed to learn semantically meaningful focus regions, such as operators, exponents, and nested mathematical notation, without requiring any supervision. The progressive masking curriculum encourages the network to become increasingly robust to missing or occluded visual information, ultimately improving structural understanding. Our complete pipeline consists of (1) self-supervised pretraining of the encoder, (2) self-supervised attention learning, and (3) supervised fine-tuning with a transformer decoder to generate LATEX sequences. Extensive experiments on CROHME benchmarks demonstrate that our method outperforms existing SSL and fully supervised baselines, validating the effectiveness of our progressive attention mechanism in enhancing HMER performance. Our codebase can be found here.

手写识别自监督注意力机制数学表达式

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。