让AI理解模糊的遥感需求,自动选择合适分析方式。
RemoteAgent: Bridging Vague Human Intents and Earth Observation with RL-based Agentic MLLMs

- 用强化学习让大模型学会判断任务该自处理还是调工具
- 在多个遥感任务上表现优于现有方法,准确率提升12%
- 适合需要理解自然语言指令的遥感分析用户
地球观测(EO)系统主要服务于领域专家,但用户常以模糊自然语言表达需求,而非精确指令。同一查询在不同场景下可能需要从整体图像理解到像素级预测的不同精度。尽管多模态大模型(MLLM)具备强语义理解能力,但其文本输出格式难以支持高精度空间预测。现有代理框架虽通过外部工具弥补不足,但盲目调用效率低,且未充分发挥模型自身能力。为此,我们提出RemoteAgent,一个基于强化学习的智能体框架,尊重MLLM的能力边界。为提升对真实意图的理解,我们构建了VagueEO数据集,将遥感任务与模拟的模糊自然语言查询配对。利用该数据集进行强化微调,使MLLM成为能直接处理图像和稀疏区域任务的认知核心。远程代理仅在需要密集预测时,通过模型上下文协议调用专用工具。大量实验表明,RemoteAgent在多种遥感任务中均展现出强大的意图识别能力,性能达到行业领先水平。
原文摘要 · Abstract (English)
Earth Observation (EO) systems are essentially designed to support domain experts who often express their requirements through vague natural language rather than precise, machine-friendly instructions. Depending on the specific application scenario, these vague queries can demand vastly different levels of visual precision. Consequently, a practical EO AI system must bridge the gap between ambiguous human queries and the appropriate multi-granularity visual analysis tasks, ranging from holistic image interpretation to fine-grained pixel-wise predictions. While Multi-modal Large Language Models (MLLMs) demonstrate strong semantic understanding, their text-based output format is inherently ill-suited for dense, precision-critical spatial predictions. Existing agentic frameworks address this limitation by delegating tasks to external tools, but indiscriminate tool invocation is computationally inefficient and underutilizes the MLLM's native capabilities. To this end, we propose RemoteAgent, an agentic framework that strategically respects the intrinsic capability boundaries of MLLMs. To empower this framework to understand real user intents, we construct VagueEO, a human-centric instruction dataset pairing EO tasks with simulated vague natural-language queries. By leveraging VagueEO for reinforcement fine-tuning, we align an MLLM into a robust cognitive core that directly resolves image- and sparse region-level tasks. Consequently, RemoteAgent processes suitable tasks internally while intelligently orchestrating specialized tools via the Model Context Protocol exclusively for dense predictions. Extensive experiments demonstrate that RemoteAgent achieves robust intent recognition capabilities while delivering highly competitive performance across diverse EO tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。