轻量级编码器模型实现遥感多任务高效融合
An Efficient and Effective Encoder Model for Vision and Language Tasks in the Remote Sensing Domain
- 采用纯编码器架构，减少参数量提升效率
- 支持图像生成文本与跨模态检索的联合建模
- 适合资源有限机构部署遥感多模态任务
遥感领域近年来涌现出基于大视觉语言模型（LVLM）的方法，可处理计算机视觉与自然语言处理交叉任务。为充分挖掘此类模型潜力，研究重点转向收集涵盖图像描述、视觉问答等多任务的大规模训练数据。然而，由于参数量巨大，训练和推理成本高昂，多数机构难以承担。本文探索纯编码器结构，提出一种紧凑高效的多任务学习模型——GeoMELT（Multi-task Efficient Learning Transformer），能有效处理非典型联合任务组合：从遥感图像生成文本及跨模态检索。在多个基准测试中，该模型展现出优异的性能与计算效率。
原文摘要 · Abstract (English)
The remote sensing community has recently seen the emergence of methods based on Large Vision and Language Models (LVLMs) that can address multiple tasks at the intersection of computer vision and natural language processing. To fully exploit the potential of such models, a significant focus has been given to the collection of large amounts of training data that cover multiple remote sensing-specific tasks, such as image captioning or visual question answering. However, the cost of using and training LVLMs is high, due to the large number of parameters. While multiple parameter-efficient adaptation techniques have been explored, the computational costs of training and inference with these models can remain prohibitive for most institutions. In this work, we explore the use of encoder-only architectures and propose a model that can effectively address multi-task learning while remaining compact in terms of the number of parameters. In particular, our model tackles combinations of tasks that are not typically explored in a unified model: the generation of text from remote sensing images and cross-modal retrieval. The results of our GeoMELT model - named from Multi-task Efficient Learning Transformer - in established benchmarks confirm the efficacy and efficiency of the proposed approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。