让AI理解多种提问方式,自动匹配知识点并生成解释。
From Query to Explanation: Uni-RAG for Multi-Modal Retrieval-Augmented Learning in STEM
- 用原型匹配动态检索教育内容,支持多样提问风格。
- 在多个基准上优于基线系统,生成质量高且计算成本低。
- 适合需要个性化、可解释性教学的STEM智能系统。
在人工智能辅助教学中,利用多种查询风格来解析抽象教育内容,对提供有效且可访问的学习体验至关重要。然而,现有检索系统主要聚焦自然文本-图像匹配,难以应对真实教育场景中的多样性与模糊性。为此,我们开发了一个轻量高效多模态检索模块Uni-Retrieval,通过提取查询风格原型,并与持续更新的Prompt Bank中的标记动态匹配。该Prompt Bank利用专家混合低秩适配(MoE-LoRA)模块编码存储领域知识,可适应未见查询类型。为实现自然语言教育内容生成,我们将Uni-Retrieval与小型指令微调语言模型结合,构建完整的检索增强生成框架Uni-RAG。给定风格条件查询后,Uni-RAG首先检索相关学习材料,再生成符合学习目标的人类可读解释、反馈或教学内容。在SER及其他多模态基准上的实验结果表明,Uni-RAG在检索准确率和生成质量上均优于基线检索与RAG系统,同时保持低计算开销。本框架为智能教育系统提供了可扩展、具教学基础的解决方案,弥合检索与生成,支持跨多样化STEM场景的个性化、可解释、高效学习辅助。
原文摘要 · Abstract (English)
In AI-facilitated teaching, leveraging various query styles to interpret abstract educational content is crucial for delivering effective and accessible learning experiences. However, existing retrieval systems predominantly focus on natural text-image matching and lack the capacity to address the diversity and ambiguity inherent in real-world educational scenarios. To address this limitation, we develop a lightweight and efficient multi-modal retrieval module, named Uni-Retrieval, which extracts query-style prototypes and dynamically matches them with tokens from a continually updated Prompt Bank. This Prompt Bank encodes and stores domain-specific knowledge by leveraging a Mixture-of-Expert Low-Rank Adaptation (MoE-LoRA) module and can be adapted to enhance Uni-Retrieval's capability to accommodate unseen query types at test time. To enable natural language educational content generation, we integrate the original Uni-Retrieval with a compact instruction-tuned language model, forming a complete retrieval-augmented generation pipeline named Uni-RAG. Given a style-conditioned query, Uni-RAG first retrieves relevant educational materials and then generates human-readable explanations, feedback, or instructional content aligned with the learning objective. Experimental results on SER and other multi-modal benchmarks show that Uni-RAG outperforms baseline retrieval and RAG systems in both retrieval accuracy and generation quality, while maintaining low computational cost. Our framework provides a scalable, pedagogically grounded solution for intelligent educational systems, bridging retrieval and generation to support personalized, explainable, and efficient learning assistance across diverse STEM scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。