专为月球探测设计的多模态模型,用真实月面图像和描述提升科学分析能力。
LLaVA-LE: Large Language-and-Vision Assistant for Lunar Exploration
- 构建96万张月面图像配图文数据集LUCID,支持地形特征描述与问答训练。
- 两阶段微调使模型在月面分析任务上性能比基础版高3.3倍,推理得分超人工参考。
- 适合行星科学、遥感分析及多模态大模型研究者使用。
多模态视觉-语言模型(VLMs)虽能联合处理视觉与文本信息,但其在行星科学中的应用仍鲜有探索。主要瓶颈在于缺乏大规模真实行星影像与科学描述配对的数据集。本文提出针对月球表面与次表层特征分析的专用模型LLaVA-LE。为此,我们构建了包含96,000张高分辨率黑白图像及详细描述的大型多模态月面数据集LUCID,以及基于约20,000张图像生成的81,000个问答对。在此基础上,采用两阶段训练策略:(1) 领域特定地形描述的概念对齐,(2) 指令微调的视觉问答。我们还设计了涵盖多种推理复杂度的评估基准。经GPT与Gemini裁判评估,LLaVA-LE在整体性能上较基础版LLaVA提升3.3倍,较第一阶段模型提升2.1倍,推理得分为1.070,超过裁判参考分,验证了领域专用多模态数据与指令微调的有效性。代码已开源。
原文摘要 · Abstract (English)
Recent advances in multimodal vision-language models (VLMs) have enabled joint reasoning over visual and textual information, yet their application to planetary science remains largely unexplored. A key hindrance is the absence of large-scale datasets that pair real planetary imagery with detailed scientific descriptions. In this work, we introduce LLaVA-LE (Large Language-and-Vision Assistant for Lunar Exploration), a vision-language model specialized for lunar surface and subsurface characterization. To enable this capability, we curate a new large-scale multimodal lunar dataset, LUCID (LUnar Caption Image Dataset) consisting of 96k high-resolution panchromatic images paired with detailed captions describing lunar terrain characteristics, and 81k question-answer (QA) pairs derived from approximately 20k images in the LUCID dataset. Leveraging this dataset, we fine-tune LLaVA using a two-stage training curriculum: (1) concept alignment for domain-specific terrain description, and (2) instruction-tuned visual question answering. We further design evaluation benchmarks spanning multiple levels of reasoning complexity relevant to lunar terrain analysis. Evaluated against GPT and Gemini judges, LLaVA-LE achieves a 3.3x overall performance gain over Base LLaVA and 2.1x over our Stage 1 model, with a reasoning score of 1.070, exceeding the judge's own reference score, highlighting the effectiveness of domain-specific multimodal data and instruction tuning to advance VLMs in planetary exploration. Code is available at https://github.com/OSUPCVLab/LLaVA-LE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。