用大模型让无人机和无人船自动巡检港口，懂指令还能自适应
LLM-VLM Fusion Framework for Autonomous Maritime Port Inspection using a Heterogeneous UAV-USV System
- 用大语言模型生成带约束的执行计划，实现无人机与无人船协同
- 视觉语言模型实时识别违规并生成带上下文的检查报告
- 轻量化设计适合资源受限的海上机器人，适合智能巡检场景
港口巡检在保障复杂海事环境中的安全、合规与运营效率方面至关重要。现有方法多依赖人工操作和传统计算机视觉技术，缺乏可扩展性与语境理解能力。本文提出一种融合大语言模型（LLMs）与视觉语言模型（VLMs）的集成工程框架，实现基于异构无人机-无人船系统的自主港口巡检。该框架以LLM驱动的符号化规划替代传统状态机任务规划，通过依赖图编码操作约束，确保无人机与无人船的安全协作；同时，VLM模块实现基于语义的实时巡检与合规评估，生成带有上下文推理的结构化报告。框架在扩展版MBZIRC海事模拟器上验证，涵盖真实港口基础设施，并通过真实机器人巡检实验进一步评估。轻量级机载设计使其适用于资源受限的海上平台，推动智能自主巡检系统的发展。项目代码与视频见：https://github.com/Muhayyuddin/llm-vlm-fusion-port-inspection
原文摘要 · Abstract (English)
Maritime port inspection plays a critical role in ensuring safety, regulatory compliance, and operational efficiency in complex maritime environments. However, existing inspection methods often rely on manual operations and conventional computer vision techniques that lack scalability and contextual understanding. This study introduces a novel integrated engineering framework that utilizes the synergy between Large Language Models (LLMs) and Vision Language Models (VLMs) to enable autonomous maritime port inspection using cooperative aerial and surface robotic platforms. The proposed framework replaces traditional state-machine mission planners with LLM-driven symbolic planning and improved perception pipelines through VLM-based semantic inspection, enabling context-aware and adaptive monitoring. The LLM module translates natural language mission instructions into executable symbolic plans with dependency graphs that encode operational constraints and ensure safe UAV-USV coordination. Meanwhile, the VLM module performs real-time semantic inspection and compliance assessment, generating structured reports with contextual reasoning. The framework was validated using the extended MBZIRC Maritime Simulator with realistic port infrastructure and further assessed through real-world robotic inspection trials. The lightweight on-board design ensures suitability for resource-constrained maritime platforms, advancing the development of intelligent, autonomous inspection systems. Project resources (code and videos) can be found here: https://github.com/Muhayyuddin/llm-vlm-fusion-port-inspection
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。