用视觉语言模型自动识别实验室隐患,通过结构化场景图提升检测准确率。
Toward Autonomous Laboratory Safety Monitoring with Vision Language Models: Learning to See Hazards Through Scene Structure
- 将文本场景转为图像+场景图+标注三元组,构建可视化安全数据集。
- 纯视觉输入下模型性能下降显著,说明直接从像素提取关系困难。
- 引入场景图引导对齐,提升模型在真实视觉环境中的隐患识别能力。
实验室因微小不当操作易引发严重伤害,但持续安全监控受限于人力。视觉语言模型(VLMs)有望实现自动化监控,然而其在真实场景下的表现尚不明确,因安全事件多以非结构化文本记录。为此,我们提出一种结构化数据生成流程,利用大语言模型构建场景图,图像生成模型渲染图像,形成(图像,场景图,真值)三元组。在包含1,207个样本、362种独特场景的合成数据集上,测试七种开源与闭源VLMs发现:给定文本场景图时模型表现良好,但在纯视觉设置下性能显著下降,表明直接从像素提取结构化关系存在困难。为此,我们提出后训练阶段的上下文工程方法——场景图引导对齐,将视觉输入转化为更符合VLM推理逻辑的结构化场景图,有效弥补感知差距,在纯视觉场景中提升隐患检测性能。
原文摘要 · Abstract (English)
Laboratories are prone to severe injuries from minor unsafe actions, yet continuous safety monitoring -- beyond mandatory pre-lab safety training -- is limited by human availability. Vision language models (VLMs) offer promise for autonomous laboratory safety monitoring, but their effectiveness in realistic settings is unclear due to the lack of visual evaluation data, as most safety incidents are documented primarily as unstructured text. To address this gap, we first introduce a structured data generation pipeline that converts textual laboratory scenarios into aligned triples of (image, scene graph, ground truth), using large language models as scene graph architects and image generation models as renderers. Our experiments on the synthetic dataset of 1,207 samples across 362 unique scenarios and seven open- and closed-source models show that VLMs perform effectively given textual scene graph, but degrade substantially in visual-only settings indicating difficulty in extracting structured object relationships directly from pixels. To overcome this, we propose a post-training context-engineering approach, scene-graph-guided alignment, to bridge perceptual gaps in VLMs by translating visual inputs into structured scene graphs better aligned with VLM reasoning, improving hazard detection performance in visual only settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。