用视觉Transformer将手写公式转为LaTeX，效果优于传统方法
Automated LaTeX Code Generation from Handwritten Math Expressions Using Vision Transformer
- 用Vision Transformer替代传统CNN+RNN结构
- 在准确率和BLEU分数上均超越基线模型
- 适合做数学公式识别与自动化写作的研究者
将手写或数字格式的数学表达式转换为LaTeX代码是一项重大挑战。本文研究了基于Transformer架构的方法，用于将数学表达式图像转换为对应的LaTeX代码。作为基线，采用当前最先进的CNN编码器与LSTM解码器。此外，通过将CNN编码器替换为预训练的ResNet50模型（适配灰度输入）来改进CNN-RNN结构，并进一步实验了视觉Transformer模型，与基线及CNN-LSTM模型进行对比。结果表明，视觉Transformer架构在整体准确率、BLEU得分上均优于基线的CNN-RNN框架，同时达到更低的莱文斯坦距离。这些结果也表明，通过调整模型参数可实现进一步提升。为促进开放研究，我们公开了模型实现，支持结果复现并推动该领域的后续研究。
原文摘要 · Abstract (English)
Transforming mathematical expressions into LaTeX poses a significant challenge. In this paper, we examine the application of advanced transformer-based architectures to address the task of converting handwritten or digital mathematical expression images into corresponding LaTeX code. As a baseline, we utilize the current state-of-the-art CNN encoder and LSTM decoder. Additionally, we explore enhancements to the CNN-RNN architecture by replacing the CNN encoder with the pretrained ResNet50 model with modification to suite the grey scale input. Further, we experiment with vision transformer model and compare with Baseline and CNN-LSTM model. Our findings reveal that the vision transformer architectures outperform the baseline CNN-RNN framework, delivering higher overall accuracy and BLEU scores while achieving lower Levenshtein distances. Moreover, these results highlight the potential for further improvement through fine-tuning of model parameters. To encourage open research, we also provide the model implementation, enabling reproduction of our results and facilitating further research in this domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。