让海洋科学家用自然语言直接查询海量海洋图像数据。
FathomGPT: A Natural Language Interface for Interactively Exploring Ocean Science Data
- 基于大模型构建自然语言接口，支持复杂查询与图像搜索。
- 可将常见名称映射为科学名，实时生成交互图表。
- 专为海洋科研设计，适合非编程背景的科学家使用。
我们提出FathomGPT，一个开源系统，通过自然语言接口实现对海洋科学数据的交互式探索。该系统与海洋科学家紧密合作，用于浏览和分析FathomNet图像数据库。FathomGPT提供定制的信息检索管道，利用OpenAI的大语言模型，支持：创建复杂查询以检索图像、分类信息及科学测量数据；将通用名称和形态特征映射为科学名称；按需生成交互式图表；以及通过图像或指定模式进行搜索。在设计中，特别注重提升用户体验，支持自由探索并优化响应速度。本文介绍FathomGPT的架构与实现细节，并通过一系列消融实验验证了名称解析、微调和提示工程的有效性。同时展示了交互式数据探索的应用场景，并记录了海洋科学家与机器学习专家的反馈。
原文摘要 · Abstract (English)
We introduce FathomGPT, an open source system for the interactive investigation of ocean science data via a natural language interface. FathomGPT was developed in close collaboration with marine scientists to enable researchers to explore and analyze the FathomNet image database. FathomGPT provides a custom information retrieval pipeline that leverages OpenAI's large language models to enable: the creation of complex queries to retrieve images, taxonomic information, and scientific measurements; mapping common names and morphological features to scientific names; generating interactive charts on demand; and searching by image or specified patterns within an image. In designing FathomGPT, particular emphasis was placed on enhancing the user's experience by facilitating free-form exploration and optimizing response times. We present an architectural overview and implementation details of FathomGPT, along with a series of ablation studies that demonstrate the effectiveness of our approach to name resolution, fine tuning, and prompt modification. We also present usage scenarios of interactive data exploration sessions and document feedback from ocean scientists and machine learning experts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。