对比RAG与GraphRAG在数学教材页级问答中的表现
Comparing RAG and GraphRAG for Page-Level Retrieval Question Answering on a Math Textbook
- 用嵌入模型和图结构检索，比对页面定位与答案质量
- RAG在99.4%准确率下胜过GraphRAG，BM25也表现强劲
- 适合资源有限但需精准引用教材的教育AI系统
大型语言模型（LLMs）在教育辅助中潜力巨大，但常与特定课程材料脱节。本文研究了检索增强生成（RAG）与GraphRAG在本科数学教材页级问答中的表现。基于包含477个问题-答案对的标注数据集（每对关联特定教材页），比较五种基于嵌入的RAG模型、一个BM25基线与GraphRAG在两个指标上的表现：检索准确率（是否正确检索到目标页）与答案质量（F1得分）。结果表明，基于嵌入的RAG在页级检索上优于GraphRAG，voyage-3-large在top-10时达到99.4%准确率（bootstrap 95%置信区间：top-1为[.644, .728]）。BM25作为基线表现优异，超过多个嵌入模型。错误分析显示，63.3%的top-1失败案例仍检索到同章节内容，表明即使失败也具教学相关性。GraphRAG检索上下文过多（约47K tokens，而RAG约为3.7K），导致生成质量下降。进一步使用开源本地LLM（Qwen3.5-35B-A3B）复现关键实验发现，对于较弱模型，RAG带来的性能提升更大（相对F1提升+39%对比+16%），这对成本敏感的教育部署具有重要意义。这些发现为参考特定教材页的AI辅导系统设计提供了依据。
原文摘要 · Abstract (English)
Large language models (LLMs) show promise as educational aids but often lack alignment with specific course materials. We investigate Retrieval-Augmented Generation (RAG) and GraphRAG for page-level question answering on an undergraduate mathematics textbook. Using a curated dataset of 477 question-answer pairs, each tied to a specific textbook page, we compare five embedding-based RAG models, a BM25 baseline, and GraphRAG across two metrics: retrieval accuracy (whether the correct page is retrieved) and answer quality (F1 score). Our results show that embedding-based RAG outperforms GraphRAG for page-level retrieval, with voyage-3-large achieving 99.4% accuracy at top-10 (bootstrap 95% CI for top-1: [.644, .728]). BM25 proves a strong baseline, outperforming several embedding models. Error analysis reveals that 63.3% of top-1 failures retrieve same-chapter content, suggesting pedagogical relevance even in failure cases. GraphRAG retrieves excessive context (~47K tokens vs. ~3.7K for RAG), reducing generation quality. We further replicate key experiments using an open-source local LLM (Qwen3.5-35B-A3B), finding that RAG benefits are proportionally larger for weaker models (+39% vs. +16% relative F1 improvement), an important result for cost-sensitive educational deployments. These findings inform the design of AI tutoring systems that reference specific textbook pages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。