首个自由探索式家庭安全检测基准,评估视觉语言模型的隐患发现能力。
HomeSafeBench: Benchmarking Embodied Vision-Language Models in Free-Exploration Home Safety Inspection
- 构建虚拟家庭环境,让智能体通过第一视角自主巡查并报告隐患。
- 顶尖模型仅达34.7%准确率,远低于人类98.0%,且漏报严重。
- 提出回溯线索法,低成本提升模型对隐患的识别能力。
家庭安全隐患是导致可预防居家伤害的主要原因,推动了自动化巡检系统的研发。我们提出HomeSafeBench,首个基于第一人称视觉反馈的自由探索式家庭安全检测基准,其基于VirtualHome模拟器,涵盖五类常见家庭隐患,包含1,000个经人工验证的检测任务。该基准允许具身智能体在全交互3D环境中导航、调整视角,并仅通过渲染的第一人称画面报告隐患。评估多种前沿视觉-语言模型(VLMs)发现,最优模型仅达到约34.7% F1值,远低于人类98.0%水平;且所有模型均呈现高精度、低召回现象,表明风险识别存在系统性缺陷。为低成本填补差距,我们提出CueBack:一种离线数据构造方法,利用检查中‘线索先于确认’的结构,从优势轨迹回溯至隐患线索首次可见帧,并重构为可执行监督信号。在4B规模的VLM上微调后,跨分布测试集平均F1从18.7%提升至45.3%,超越最强闭源模型性能。基准、训练数据与代码已开源。
原文摘要 · Abstract (English)
Safety hazards in the home are a leading cause of preventable domestic injuries, motivating an automated inspector that actively explores a home and reports hazards before they cause harm. We introduce HomeSafeBench, the first benchmark for free-exploration home safety inspection with egocentric visual feedback, in which an embodied agent navigates a fully interactive 3D home, adjusts its viewpoint, and reports hazards purely from rendered first-person views. Built on the VirtualHome simulator, it covers five categories of common household hazards and comprises 1,000 human-validated inspection tasks. Evaluating a broad range of state-of-the-art Vision-Language Models (VLMs) reveals a large gap, where the best model reaches only about 34.7% F1, far below the 98.0% of a human inspector. Moreover, precision far exceeds recall across models, revealing a systematic tendency to under-report hazards that reflects a shared deficiency in risk recognition. To close this gap at low cost, we propose CueBack, an offline data-construction method that exploits the clue-precedes-confirmation structure of inspection, backtracking a privileged trajectory to the earliest frame where a hazard cue becomes visible and rewriting it into executable supervision. Fine-tuning a 4B-size VLM on CueBack-constructed data raises the average F1 from 18.7% to 45.3% on an out-of-distribution test set, surpassing the strongest closed-source model performance 34.7%. The benchmark, training dataset, and code are available at https://github.com/BITHLP/HomeSafeBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。