用检索增强生成提升AI教学可靠性,对比两种方法优劣。
Aligning LLMs for the Classroom with Knowledge-Based Retrieval -- A Comparative RAG Study
- 对比向量与图结构检索,适配不同教学场景
- 图检索在主题问答中表现更优,向量检索成本更低
- 提出动态路由框架,兼顾准确与效率
大型语言模型如ChatGPT在课堂中日益普及,但常提供过时或虚构信息,可能误导学生。检索增强生成(RAG)通过外部资源支撑回答,提升可靠性。本文研究两种易实现的RAG范式:基于向量的检索与基于图的检索,旨在探索课堂问答的最佳实践。现有比较研究未考虑教育学科、问题类型及部署成本等教学因素。我们使用新构建的数据集EduScopeQA(涵盖3,176道跨学科题目),评估多种教育类查询的表现,包括具体事实与广泛主题讨论。同时,通过系统性修改教科书的数据集,测试模型对非一致知识的适应能力。结果表明,OpenAI向量搜索RAG作为低成本通用方案,在快速获取事实方面表现良好;而GraphRAG Global在主题类问题上提供更具教学价值的回答,GraphRAG Local在教科书被密集篡改的场景下达到最高准确率。考虑到图检索资源消耗为向量检索的10-20倍,本文提出动态分支框架,根据问题类型自动选择最优检索方式,显著提升回答可信度与运行效率。研究为教育者与系统设计者提供了可落地的集成指导。
原文摘要 · Abstract (English)
Large language models like ChatGPT are increasingly used in classrooms, but they often provide outdated or fabricated information that can mislead students. Retrieval Augmented Generation (RAG) improves reliability of LLMs by grounding responses in external resources. We investigate two accessible RAG paradigms, vector-based retrieval and graph-based retrieval to identify best practices for classroom question answering (QA). Existing comparative studies fail to account for pedagogical factors such as educational disciplines, question types, and practical deployment costs. Using a novel dataset, EduScopeQA, of 3,176 questions across academic subjects, we measure performance on various educational query types, from specific facts to broad thematic discussions. We also evaluate system alignment with a dataset of systematically altered textbooks that contradict the LLM's latent knowledge. We find that OpenAI Vector Search RAG (representing vector-based RAG) performs well as a low-cost generalist, especially for quick fact retrieval. On the other hand, GraphRAG Global excels at providing pedagogically rich answers to thematic queries, and GraphRAG Local achieves the highest accuracy with the dense, altered textbooks when corpus integrity is critical. Accounting for the 10-20x higher resource usage of GraphRAG (representing graph-based RAG), we show that a dynamic branching framework that routes queries to the optimal retrieval method boosts fidelity and efficiency. These insights provide actionable guidelines for educators and system designers to integrate RAG-augmented LLMs into learning environments effectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。