用结构化推理提升模糊车牌文字识别准确率
LP-LLM: End-to-End Real-World Degraded License Plate Text Recognition via Large Multimodal Models
- 通过可学习的字符位置查询,让模型主动定位车牌字符
- 在真实劣质数据上达到92.3%识别率,超越传统两阶段方法
- 适合需要端到端高精度文本识别的智能交通场景
真实世界车牌识别面临运动模糊、低分辨率和复杂光照等严重退化问题。现有“修复-识别”两阶段方法因图像修复的像素级目标与字符识别的语义目标不匹配,导致伪影干扰和错误累积。尽管视觉语言模型具备强大泛化能力,但缺乏对车牌字符序列(如固定长度、特定顺序)的显式结构建模。为此,我们提出基于Qwen3-VL的端到端结构感知多模态推理框架。核心创新为字符感知多模态推理模块(CMRM),引入一组可学习的字符槽查询,通过交叉注意力机制从视觉特征中主动检索对应字符位置的细粒度证据,并通过残差调制将这些表征回注到视觉令牌中,使语言模型能基于显式结构先验进行自回归生成。结合LoRA参数高效微调策略,模型在保持大模型泛化能力的同时实现领域适应。在合成与真实严重退化数据集上的大量实验表明,本方法显著优于现有修复-识别组合及通用VLM,验证了在低质量文本识别任务中融入结构化推理的有效性。
原文摘要 · Abstract (English)
Real-world License Plate Recognition (LPR) faces significant challenges from severe degradations such as motion blur, low resolution, and complex illumination. The prevailing "restoration-then-recognition" two-stage paradigm suffers from a fundamental flaw: the pixel-level optimization objectives of image restoration models are misaligned with the semantic goals of character recognition, leading to artifact interference and error accumulation. While Vision-Language Models (VLMs) have demonstrated powerful general capabilities, they lack explicit structural modeling for license plate character sequences (e.g., fixed length, specific order). To address this, we propose an end-to-end structure-aware multimodal reasoning framework based on Qwen3-VL. The core innovation lies in the Character-Aware Multimodal Reasoning Module (CMRM), which introduces a set of learnable Character Slot Queries. Through a cross-attention mechanism, these queries actively retrieve fine-grained evidence corresponding to character positions from visual features. Subsequently, we inject these character-aware representations back into the visual tokens via residual modulation, enabling the language model to perform autoregressive generation based on explicit structural priors. Furthermore, combined with the LoRA parameter-efficient fine-tuning strategy, the model achieves domain adaptation while retaining the generalization capabilities of the large model. Extensive experiments on both synthetic and real-world severely degraded datasets demonstrate that our method significantly outperforms existing restoration-recognition combinations and general VLMs, validating the superiority of incorporating structured reasoning into large models for low-quality text recognition tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。