用小模型+目标检测,让工地安全隐患识别更准更快。
Integration of Object Detection and Small VLMs for Construction Safety Hazard Identification
- 用YOLOv11n定位工人和机械,生成结构化提示引导小模型推理。
- 最佳模型F1提升至50.6%(基线34.5%),解释质量显著提高。
- 轻量级设计仅增加2.5ms延迟,适合实时工地安全监控。
准确及时地识别工人周围的施工隐患对预防事故至关重要。尽管大视觉语言模型(VLMs)具备强大的上下文推理能力,但其高计算需求限制了在近实时施工隐患检测中的应用。相比之下,参数少于40亿的小型视觉语言模型(sVLMs)虽效率更高,但在复杂施工场景下常出现准确率下降和幻觉问题。为此,本研究提出一种检测引导的sVLM框架,将目标检测与多模态推理结合,实现上下文感知的隐患识别。该框架首先使用YOLOv11n检测场景中的工人和施工机械,将其检测结果嵌入结构化提示以引导sVLM的推理过程,实现空间定位的隐患评估。在包含隐患标注和解释理由的施工场景图像数据集上,对六种sVLM(Gemma-3 4B、Qwen-3-VL 2B/4B、InternVL-3 1B/2B、SmolVLM-2B)进行了零样本测试。所提方法在所有模型上均提升了隐患检测性能,其中表现最佳的Gemma-3 4B达到50.6% F1-score,优于基线的34.5%。解释质量也明显改善,BERTScore F1从0.61升至0.82。尽管引入目标检测,框架推理时仅增加2.5ms/图像开销。结果表明,将轻量级目标检测与小VLM推理结合,是实现上下文感知施工安全检测的有效高效方案。
原文摘要 · Abstract (English)
Accurate and timely identification of construction hazards around workers is essential for preventing workplace accidents. While large vision-language models (VLMs) demonstrate strong contextual reasoning capabilities, their high computational requirements limit their applicability in near real-time construction hazard detection. In contrast, small vision-language models (sVLMs) with fewer than 4 billion parameters offer improved efficiency but often suffer from reduced accuracy and hallucination when analyzing complex construction scenes. To address this trade-off, this study proposes a detection-guided sVLM framework that integrates object detection with multimodal reasoning for contextual hazard identification. The framework first employs a YOLOv11n detector to localize workers and construction machinery within the scene. The detected entities are then embedded into structured prompts to guide the reasoning process of sVLMs, enabling spatially grounded hazard assessment. Within this framework, six sVLMs (Gemma-3 4B, Qwen-3-VL 2B/4B, InternVL-3 1B/2B, and SmolVLM-2B) were evaluated in zero-shot settings on a curated dataset of construction site images with hazard annotations and explanatory rationales. The proposed approach consistently improved hazard detection performance across all models. The best-performing model, Gemma-3 4B, achieved an F1-score of 50.6%, compared to 34.5% in the baseline configuration. Explanation quality also improved significantly, with BERTScore F1 increasing from 0.61 to 0.82. Despite incorporating object detection, the framework introduces minimal overhead, adding only 2.5 ms per image during inference. These results demonstrate that integrating lightweight object detection with small VLM reasoning provides an effective and efficient solution for context-aware construction safety hazard detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。