用视觉语言模型统一识别车牌，准确率提升14%以上。
Neural Sentinel: Unified Vision Language Model (VLM) for License Plate Recognition with Human-in-the-Loop Continual Learning
- 用PaliGemma模型+LoRA微调，单次推理完成车牌、状态、属性识别
- 92.3%识别准确率，比EasyOCR高14.1%，延迟仅152ms
- 支持人工反馈持续学习，零样本泛化到颜色、安全带等任务
传统车牌识别系统采用检测+OCR的多阶段流水线，导致误差累积、延迟高且结构复杂。本文提出Neural Sentinel，一种基于视觉语言模型（VLM）的统一方法，通过单次前向传播实现车牌识别、状态分类与车辆属性提取。主要贡献在于证明经低秩适配（LoRA）微调的PaliGemma 3B模型可同时回答多个视觉问题，达到92.3%的车牌识别准确率，较EasyOCR提升14.1%，较PaddleOCR提升9.9%。引入人机协同持续学习框架，通过经验回放防止灾难性遗忘，维持原始数据与修正样本70:30比例。系统平均推理延迟为152ms，预期校准误差（ECE）为0.048，表明置信度估计良好。此外，该VLM架构具备零样本泛化能力，无需特定训练即可完成车辆颜色检测（89%）、安全带检测（82%）和乘员计数（78%）。在真实收费站图像上大量实验表明，统一视觉语言范式为车牌识别带来范式转变，兼具更高精度、更低复杂度和涌现多任务能力。
原文摘要 · Abstract (English)
Traditional Automatic License Plate Recognition (ALPR) systems employ multi-stage pipelines consisting of object detection networks followed by separate Optical Character Recognition (OCR) modules, introducing compounding errors, increased latency, and architectural complexity. This research presents Neural Sentinel, a novel unified approach that leverages Vision Language Models (VLMs) to perform license plate recognition, state classification, and vehicle attribute extraction through a single forward pass. Our primary contribution lies in demonstrating that a fine-tuned PaliGemma 3B model, adapted via Low-Rank Adaptation (LoRA), can simultaneously answer multiple visual questions about vehicle images, achieving 92.3% plate recognition accuracy, which is a 14.1% improvement over EasyOCR and 9.9% improvement over PaddleOCR baselines. We introduce a Human-in-the-Loop (HITL) continual learning framework that incorporates user corrections while preventing catastrophic forgetting through experience replay, maintaining a 70:30 ratio of original training data to correction samples. The system achieves a mean inference latency of 152ms with an Expected Calibration Error (ECE) of 0.048, indicating well calibrated confidence estimates. Additionally, the VLM first architecture enables zero-shot generalization to auxiliary tasks including vehicle color detection (89%), seatbelt detection (82%), and occupancy counting (78%) without task specific training. Through extensive experimentation on real world toll plaza imagery, we demonstrate that unified vision language approaches represent a paradigm shift in ALPR systems, offering superior accuracy, reduced architectural complexity, and emergent multi-task capabilities that traditional pipeline approaches cannot achieve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。