用大模型直接像人一样画建筑轮廓,精度和泛化能力都更强。
VectorLLM: Human-like Extraction of Structured Building Contours vis Multimodal LLMs
- 模仿人工标注,逐角点回归建筑边界
- 在三个数据集上分别提升5.6、7.1、13.6的精度
- 零样本识别飞机、水体等新目标,适合多场景遥感应用
从遥感影像中自动提取矢量化建筑轮廓对城市规划、人口估算和灾害评估至关重要。现有最先进方法依赖复杂的多阶段流程,包括像素分割、矢量化和多边形优化,限制了其可扩展性和实际应用。受大语言模型(LLM)强大推理能力启发,我们提出VectorLLM,首个专为遥感图像建筑轮廓提取设计的多模态大模型(MLLM)。与以往方法不同,VectorLLM直接逐角点回归建筑轮廓,模拟人工标注过程。其架构包含视觉基础主干、MLP连接器和大语言模型,并引入可学习位置嵌入以增强空间理解能力。通过在WHU、WHU-Mix和CrowdAI数据集上综合探索预训练、监督微调及偏好优化等训练策略,VectorLLM在三个数据集上分别领先先前最先进方法5.6、7.1、13.6的平均精度(AP)。令人瞩目的是,VectorLLM在未见物体(如飞机、水体、油罐)上展现出强大零样本性能,凸显其统一建模多种遥感目标轮廓提取任务的潜力。本工作建立了一种新的遥感矢量提取范式,利用LLM的拓扑推理能力,在实现高精度的同时具备卓越泛化性。代码与权重将公开,促进社区发展。
原文摘要 · Abstract (English)
Automatically extracting vectorized building contours from remote sensing imagery is crucial for urban planning, population estimation, and disaster assessment. Current state-of-the-art methods rely on complex multi-stage pipelines involving pixel segmentation, vectorization, and polygon refinement, which limits their scalability and real-world applicability. Inspired by the remarkable reasoning capabilities of Large Language Models (LLMs), we introduce VectorLLM, the first Multi-modal Large Language Model (MLLM) designed for regular building contour extraction from remote sensing images. Unlike existing approaches, VectorLLM performs corner-point by corner-point regression of building contours directly, mimicking human annotators' labeling process. Our architecture consists of a vision foundation backbone, an MLP connector, and an LLM, enhanced with learnable position embeddings to improve spatial understanding capability. Through comprehensive exploration of training strategies including pretraining, supervised fine-tuning, and preference optimization across WHU, WHU-Mix, and CrowdAI datasets, VectorLLM significantly outperformed the previous SOTA methods by 5.6 AP, 7.1 AP, 13.6 AP, respectively in the three datasets. Remarkably, VectorLLM exhibits strong zero-shot performance on unseen objects including aircraft, water bodies, and oil tanks, highlighting its potential for unified modeling of diverse remote sensing object contour extraction tasks. Overall, this work establishes a new paradigm for vector extraction in remote sensing, leveraging the topological reasoning capabilities of LLMs to achieve both high accuracy and exceptional generalization. All the codes and weights will be published for promoting community development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。