arXiv:2508.17102cs.CV2025-08被引 4

用强化学习替代标注,让模型更懂自然语言地理图像指令。

GRASP: Geospatial pixel Reasoning viA Structured Policy learning

  • 用结构化策略学习融合大模型与分割模型,分步推理
  • 在域外场景提升54%性能,减少对密集标注依赖
  • 适合需要低标注成本、强泛化能力的遥感任务

地理空间像素推理旨在直接根据自然语言指令生成遥感图像的分割掩码。现有方法多采用监督微调范式,以密集像素级掩码为真值训练多模态大语言模型,虽在训练数据分布内有效,但面临两大问题:(1) 大规模密集标注成本高昂;(2) 监督微调在域外场景泛化能力有限。为此,我们提出GRASP,一种级联式结构化策略学习框架,将多模态大语言模型与预训练分割模型结合。为增强泛化性,引入PRIME训练范式,以强化学习替代监督微调,更好对齐推理与定位行为。为降低标注成本,设计BoP-Rewards,用边界框和正点代替密集掩码标签,并通过格式(约束输出结构可解析)和准确率(评估预测框与点质量)双重信号验证输出。评估中,我们在EarthReason和GeoPixInstruct上训练模型,合并测试集构建域内基准;进一步发布GRASP-1k,一个完全域外的基准,包含推理密集型查询、推理轨迹和细粒度掩码。实验表明,该方法在域内达当前最优,在域外场景提升高达54%,证实了带成本感知奖励的强化学习是地理空间像素推理的鲁棒且可扩展范式。代码与数据集将公开。

原文摘要 · Abstract (English)

Geospatial pixel reasoning aims to generate segmentation masks in remote sensing imagery directly from natural-language instructions. Most existing approaches follow a paradigm that fine-tunes multimodal large language models under supervision with dense pixel-level masks as ground truth. While effective within the training data distribution, this design suffers from two main drawbacks: (1) the high cost of large-scale dense mask annotation, and (2) the limited generalization capability of supervised fine-tuning in out-of-domain scenarios. To address these issues, we propose GRASP, a structured policy-learning framework that integrates a multimodal large language model with a pretrained segmentation model in a cascaded manner. To enhance generalization, we introduce PRIME, a training paradigm that replaces supervised fine-tuning with reinforcement learning to better align reasoning and grounding behaviors with task objectives. To reduce annotation costs, we design BoP-Rewards, which substitutes dense mask labels with bounding box and positive points. It further verifies outputs through two complementary signals: format, which constrains the reasoning and grounding structure to remain syntactically parsable, and accuracy, which evaluates the quality of predicted boxes and points. For evaluation, we train our method and all baselines on EarthReason and GeoPixInstruct, constructing an in-domain benchmark by merging their test sets. We further release GRASP-1k, a fully out-of-domain benchmark with reasoning-intensive queries, reasoning traces, and fine-grained masks. Experimental results demonstrate state-of-the-art (SOTA) in-domain performance and up to 54\% improvement in out-of-domain scenarios, confirming that reinforcement learning with cost-aware rewards provides a robust and scalable paradigm for geospatial pixel reasoning. All code and datasets will be released publicly.

遥感图像自然语言强化学习少样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。