arXiv:2507.09693cs.CV2025-07中稿 · ACM MM 2025被引 3

用AI自动生成跨学科实验解说,提升教学效率。

ExpStar: Towards Automatic Commentary Generation for Multi-discipline Scientific Experiments

  • 基于检索增强机制,动态调用外部知识生成解说。
  • 在7000+步级解说数据上,超越14个主流大模型。
  • 适合教育科技、AI助教及实验自动化领域使用。

实验解说在描述操作流程、揭示科学原理和融入安全规范方面至关重要。现实中,教师需依赖专业经验并投入大量时间准备此类内容。为此,我们提出跨学科科学实验的自动解说生成任务。尽管大模态模型在视频理解与推理方面取得进展,但其生成精细且深刻的实验解说能力仍待探索。本文贡献包括:(i) 构建首个面向实验解说生成的 extit{ExpInstruct} 数据集,涵盖21个科学主题、3个核心学科(科学、医疗、工程),包含超7000条步骤级解说,每条含操作描述、潜在科学原理(如化学方程式、物理定律)和安全指南;(ii) 提出 ExpStar 模型,采用检索增强机制,自适应地访问、评估并利用外部知识;(iii) 大量实验证明,ExpStar 显著优于14个领先的大模态模型,凸显数据集与模型的优势。我们认为 ExpStar 在推进AI辅助科学实验教学方面具有巨大潜力。

原文摘要 · Abstract (English)

Experiment commentary is crucial in describing the experimental procedures, delving into underlying scientific principles, and incorporating content-related safety guidelines. In practice, human teachers rely heavily on subject-specific expertise and invest significant time preparing such commentary. To address this challenge, we introduce the task of automatic commentary generation across multi-discipline scientific experiments. While recent progress in large multimodal models (LMMs) has demonstrated promising capabilities in video understanding and reasoning, their ability to generate fine-grained and insightful experiment commentary remains largely underexplored. In this paper, we make the following contributions: (i) We construct \textit{ExpInstruct}, the first dataset tailored for experiment commentary generation, featuring over 7\textit{K} step-level commentaries across 21 scientific subjects from 3 core disciplines (\ie, science, healthcare and engineering). Each sample includes procedural descriptions along with potential scientific principles (\eg, chemical equations and physical laws) and safety guidelines. (ii) We propose ExpStar, an automatic experiment commentary generation model that leverages a retrieval-augmented mechanism to adaptively access, evaluate, and utilize external knowledge. (iii) Extensive experiments show that our ExpStar substantially outperforms 14 leading LMMs, which highlights the superiority of our dataset and model. We believe that ExpStar holds great potential for advancing AI-assisted scientific experiment instruction.

实验生成多模态AI教育

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。