用统一框架提升手写公式识别，性能超越顶尖模型。
Uni-MuMER: Unified Multi-Task Fine-Tuning of Vision-Language Model for Handwritten Mathematical Expression Recognition
- 不改架构，全量微调视觉语言模型注入领域知识。
- 在CROHME和HME100K上比SSAN高16.31%，比Gemini2.5-flash高24.42%。
- 适合需要高精度公式识别的研究者与工业应用开发者。
手写数学表达式识别（HMER）因符号布局自由和书写风格多样，仍是光学字符识别（OCR）中的长期难题。以往方法多通过孤立的结构修改提升性能，难以融入统一框架。近期预训练视觉语言模型（VLM）展现出强跨任务泛化能力，为构建统一解决方案提供了可能。本文提出Uni-MuMER，无需修改模型架构，通过全量微调将领域知识注入通用框架。方法融合三项数据驱动任务：树状感知思维链（Tree-CoT）用于结构化空间推理，错误驱动学习（EDL）减少视觉相似字符混淆，符号计数（SC）提升长表达式识别一致性。在CROHME和HME100K数据集上的实验表明，Uni-MuMER达到超 state-of-the-art 性能，在零样本设置下分别超越轻量级专用模型SSAN 16.31% 和顶级VLM Gemini2.5-flash 24.42%。相关数据集、模型与代码已开源。
原文摘要 · Abstract (English)
Handwritten Mathematical Expression Recognition (HMER) remains a persistent challenge in Optical Character Recognition (OCR) due to the inherent freedom of symbol layouts and variability in handwriting styles. Prior methods have faced performance bottlenecks by proposing isolated architectural modifications, making them difficult to integrate coherently into a unified framework. Meanwhile, recent advances in pretrained vision-language models (VLMs) have demonstrated strong cross-task generalization, offering a promising foundation for developing unified solutions. In this paper, we introduce Uni-MuMER, which fully fine-tunes a VLM for the HMER task without modifying its architecture, effectively injecting domain-specific knowledge into a generalist framework. Our method integrates three data-driven tasks: Tree-Aware Chain-of-Thought (Tree-CoT) for structured spatial reasoning, Error-Driven Learning (EDL) for reducing confusion among visually similar characters, and Symbol Counting (SC) for improving recognition consistency in long expressions. Experiments on the CROHME and HME100K datasets show that Uni-MuMER achieves super state-of-the-art performance, outperforming the best lightweight specialized model SSAN by 16.31\% and the top-performing VLM Gemini2.5-flash by 24.42\% under zero-shot setting. Our datasets, models, and code are open-sourced at: {https://github.com/BFlameSwift/Uni-MuMER
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。