用视觉语言模型融合深度与语义信息,让无人机自动识别安全包裹投放点。
See&Say: Vision Language Guided Safe Zone Detection for Autonomous Package Delivery Drones
- 结合单目深度梯度与开放词汇检测,生成动态安全地图。
- 在多个阈值下均优于基线,准确率和交并比显著提升。
- 适合城市复杂环境下的无人机配送系统研发者参考。
自主无人机配送系统快速发展,但在杂乱的城市和郊区环境中,准确识别合适的包裹投放区域仍具挑战性。现有方法通常依赖几何分析或语义分割单一方式,缺乏集成语义推理能力。为此,我们提出See&Say框架,将几何安全线索与语义感知相结合,由视觉语言模型(VLM)驱动迭代优化。系统融合单目深度梯度与开放词汇检测掩码生成安全图,同时利用VLM动态调整物体类别提示并持续优化危险检测,实现动态条件下的可靠决策。当主投放点被占用或不安全时,系统可识别替代候选区域。我们构建了包含移动物体和人类活动的城市配送场景数据集进行评估。实验表明,See&Say在安全地图预测的准确率与交并比上均优于所有基线,并在多阈值下表现更优,验证了VLM引导的分割-深度融合对推动安全实用无人机配送的潜力。
原文摘要 · Abstract (English)
Autonomous drone delivery systems are rapidly advancing, but ensuring safe and reliable package drop-offs remains highly challenging in cluttered urban and suburban environments where accurately identifying suitable package drop zones is critical. Existing approaches typically rely on either geometry-based analysis or semantic segmentation alone, but these methods lack the integrated semantic reasoning required for robust decision-making. To address this gap, we propose See&Say, a novel framework that combines geometric safety cues with semantic perception, guided by a Vision-Language Model (VLM) for iterative refinement. The system fuses monocular depth gradients with open-vocabulary detection masks to produce safety maps, while the VLM dynamically adjusts object category prompts and refines hazard detection across time, enabling reliable reasoning under dynamic conditions during the final delivery phase. When the primary drop-pad is occupied or unsafe, the proposed See&Say also identifies alternative candidate zones for package delivery. We curated a dataset of urban delivery scenarios with moving objects and human activities to evaluate the approach. Experimental results show that See&Say outperforms all baselines, achieving the highest accuracy and IoU for safety map prediction as well as superior performance in alternative drop zone evaluation across multiple thresholds. These findings highlight the promise of VLM-guided segmentation-depth fusion for advancing safe and practical drone-based package delivery.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。