首个专为遥感设计的原生多模态模型，让图像与文本在统一序列中直接交互。
SkyNative: A Native Multimodal Architecture for Remote Sensing Vision-Language Understanding
- 将遥感图像转为视觉令牌，与文本令牌共存于共享序列中，实现跨模态直接交互。
- 在HRRSD、RSHR推理和OmniEarth上分别取得68.93%、47.40%、63.43%的准确率。
- 适合需要精细空间推理与多尺度目标感知的遥感分析任务。
遥感视觉语言模型（RS-VLM）通常采用预训练视觉编码器与投影模块，将图像特征映射至大语言模型的词元空间。尽管有效，这种模块化设计使视觉表征与语言推理分离，可能削弱细粒度视觉证据在复杂空间推理中的作用。该问题在覆盖广阔地理区域、包含多尺度物体、密集目标分布与复杂空间布局的遥感影像中尤为显著。本文提出SkyNative，首个探索遥感视觉语言任务原生多模态架构的研究。SkyNative通过轻量级块嵌入模块将遥感图像转换为视觉词元，并将其与文本词元置于共享自回归序列中，使文本词元可直接访问前序视觉上下文。为适应双模态异质性，引入模态感知解耦机制，对两种模态分别应用模态特定的投影、归一化与前馈变换，同时通过共享因果自注意力处理。大量实验表明，SkyNative在密集小目标感知、大尺寸上下文理解、复杂推理与鲁棒性方面表现优异，在HRRSD、RSHR推理与OmniEarth上的准确率分别为68.93%、47.40%与63.43%。结果表明，所探索的原生VLM架构为遥感视觉语言建模提供了有前景的新路径。
原文摘要 · Abstract (English)
Remote sensing vision-language models (RS-VLMs) commonly employ a pretrained vision encoder and a projection module to map image features into the token space of a large language model. Although effective, this modular RS-VLMs separates visual representation from language reasoning, potentially limiting the direct involvement of fine-grained visual evidence in complex spatial inference. This challenge is particularly relevant to remote sensing imagery, which often covers broad geographic areas and contains multi-scale objects, dense target distributions, and intricate spatial layouts. In this paper, we propose SkyNative, the first study to explore a native multimodal architecture for remote sensing vision-language tasks. SkyNative converts remote sensing images into visual tokens through a lightweight patch embedding module and places them together with text tokens in a shared autoregressive sequence, allowing textual tokens to directly access the preceding visual context. To accommodate the heterogeneous characteristics of the two modalities, we further adopt a modality-aware decoupling mechanism that applies modality-specific projections, normalization, and feed-forward transformations while processing both modalities through shared causal self-attention. Extensive experiments demonstrate SkyNative's strong capabilities in dense small-object perception, large-format contextual understanding, complex reasoning, and robustness, with scores of 68.93%, 47.40%, and 63.43% on HRRSD, RSHR reasoning, and OmniEarth, respectively. These results suggest that the native VLM architecture explored in SkyNative represents a promising approach to RS vision-language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。