arXiv:2503.04918cs.CV2025-03被引 5

微调Florence2模型,提升复杂环境下的物体检测精度。

Fine-Tuning Florence2 for Enhanced Object Detection in Un-constructed Environments: Vision-Language Model Approach

  • 用LoRA和Transformer层微调,适配非结构化场景
  • 在多个硬件上测试,mAP接近YOLOv8等主流模型
  • 适合需要视觉语言理解的现实复杂场景应用

视觉语言模型(VLMs)作为人工智能的重要工具,能融合文本与视觉信息以统一理解复杂场景。尽管基于Transformer架构的Florence2在通用任务中表现良好,但在非结构化或杂乱环境中的物体检测性能仍待深入探索。本研究针对非构造环境下的物体检测任务,对Florence2模型进行微调。实验采用NVIDIA T4、L4和A100 GPU多种硬件配置,对比AdamW、SGD优化器及不同学习率、LoRA(低秩适应)参数设置,使用代表真实无序场景的挑战性数据集进行训练与评估。优化后的Florence2模型在物体检测准确率上显著提升,平均精度均值(mAP)达到或接近YOLOv8、YOLOv9和YOLOv10等成熟模型水平。LoRA引入与Transformer层精细微调是关键增益来源。结果表明,基于Transformer的VLM如Florence2可有效适配特定领域任务,尤其在视觉复杂环境中具备与传统卷积检测器竞争的潜力,为现实世界中非结构化场景的高级视觉应用提供灵活可扩展的新范式。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) have emerged as powerful tools in artificial intelli-gence, capable of integrating textual and visual data for a unified understanding of complex scenes. While models such as Florence2, built on transformer architectures, have shown promise across general tasks, their performance in object detection within unstructured or cluttered environments remains underexplored. In this study, we fi-ne-tuned the Florence2 model for object detection tasks in non-constructed, complex environments. A comprehensive experimental framework was established involving multiple hardware configurations (NVIDIA T4, L4, and A100 GPUs), optimizers (AdamW, SGD), and varied hyperparameters including learning rates and LoRA (Low-Rank Adaptation) setups. Model training and evaluation were conducted on challenging datasets representative of real-world, disordered settings. The optimized Florence2 models exhibited significant improvements in object detection accuracy, with Mean Average Precision (mAP) metrics approaching or matching those of estab-lished models such as YOLOv8, YOLOv9, and YOLOv10. The integration of LoRA and careful fine-tuning of transformer layers contributed notably to these gains. Our find-ings highlight the adaptability of transformer-based VLMs like Florence2 for do-main-specific tasks, particularly in visually complex environments. The study under-scores the potential of fine-tuned VLMs to rival traditional convolution-based detec-tors, offering a flexible and scalable approach for advanced vision applications in re-al-world, unstructured settings.

视觉语言模型物体检测LoRA微调复杂环境

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。