用智能代理让科研资料可互动探索，支持图文混合查询。
CollEX -- A Multimodal Agentic RAG System Enabling Interactive Exploration of Scientific Collections
- 用多模态大模型做智能代理，通过聊天界面实现直观交互。
- 在32个科学数据集、6.4万条记录上验证了系统有效性。
- 适合教师、学生和研究者进行自主探究与跨学科发现。
本文提出CollEx，一种基于多模态大视觉语言模型（LVLMs）的智能检索增强生成（RAG）系统，旨在提升对大规模科学资料库的交互式探索体验。面对科学资料数量庞大且结构复杂的问题，传统搜索系统往往缺乏直观性与交互性，制约了学习者、教育工作者和研究者的使用效率。CollEx通过集成文本与视觉模态，构建可访问的智能代理，借助先进工具抽象复杂操作，使用户可通过自然对话实现好奇心驱动的探索，显著降低获取多样科学记录的门槛。该系统支持教学场景中师生自主学习，激发科学兴趣。同时助力研究人员发现跨领域关联并补充视觉数据。我们通过一个包含来自一所公立大学本地科学资料库的32个集合、超过6.4万条唯一记录的原型应用，验证了系统的有效性。
原文摘要 · Abstract (English)
In this paper, we introduce CollEx, an innovative multimodal agentic Retrieval-Augmented Generation (RAG) system designed to enhance interactive exploration of extensive scientific collections. Given the overwhelming volume and inherent complexity of scientific collections, conventional search systems often lack necessary intuitiveness and interactivity, presenting substantial barriers for learners, educators, and researchers. CollEx addresses these limitations by employing state-of-the-art Large Vision-Language Models (LVLMs) as multimodal agents accessible through an intuitive chat interface. By abstracting complex interactions via specialized agents equipped with advanced tools, CollEx facilitates curiosity-driven exploration, significantly simplifying access to diverse scientific collections and records therein. Our system integrates textual and visual modalities, supporting educational scenarios that are helpful for teachers, pupils, students, and researchers by fostering independent exploration as well as scientific excitement and curiosity. Furthermore, CollEx serves the research community by discovering interdisciplinary connections and complementing visual data. We illustrate the effectiveness of our system through a proof-of-concept application containing over 64,000 unique records across 32 collections from a local scientific collection from a public university.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。