对比GraphRAG与向量RAG在多跳需求追溯中的表现，发现结果受数据集和评估方式影响极大。
A Triple-Robustness Analysis of Retrieval-Augmented Generation for Multi-Hop Requirements Traceability
- 固定五阶段架构，变换嵌入模型、语料库和评估者进行三重鲁棒性测试
- 图谱遍历虽上下文过载，但合成器引用精准度达48%-65%，可解释先前争议
- 单个大模型评估可信度脆弱，重复判断一致性差，需警惕评估偏差
关于GraphRAG与向量RAG的评价存在分歧，且证据多基于单一语料库、嵌入模型和评估者——我们指出，这还取决于引用质量的衡量位置。本文开展三重鲁棒性分析：固定五阶段架构矩阵，分别替换嵌入模型（e5-small vs. Azure text-embedding-3-small）、语料库（DO-178C typed-edge requirements vs. MuSiQue构建的Wikipedia段落链），以及评估者（双模型GPT-5.4 vs. GPT-4.1），共执行2×4,440次主矩阵运行、600次跨语料运行及超过5,000次忠实度判断。结果显示：(C2a) GraphRAG图谱遍历导致上下文窗口过载，精度仅0.12–0.23，但合成器选择性引用精度达0.48–0.65；若将检索结果作为引用依据，则架构排名反转，部分化解了以往报告的矛盾。(C1) 答案级引用胜者依赖语料库与层级，但对嵌入模型稳健：GraphRAG在短跳DO-178C查询中持平于基线，在所有MuSiQue层级中胜出，而代理式流水线仅在3+跳查询中领先。(C2b) 忠实度具有语料库依赖性：在DO-178C上随跳跃距离下降（四个评估组合中有三个趋势显著，p<0.05）；而在Wikipedia链上，两种评估者均未出现崩溃。(C3) 单一评估者大模型忠实度极不稳定：GPT-5.4自一致性kappa仅为0.137（41%答案变更），远低于同日重测下限0.76；十一周后重评冻结输入，双方评估者kappa均≤0.14。仅使用密集嵌入训练的路由模块在跳数分类上达到宏平均F1 0.86 (C4)。我们认为，对RAG架构的宣称必须在此类鲁棒性水平下验证——包括对引用测量点的鲁棒性——方可信赖。
原文摘要 · Abstract (English)
Reported verdicts on GraphRAG versus vector RAG disagree, and the evidence is typically tied to a single corpus, embedder, and judge -- and, we show, to where citation quality is measured. We present a triple-robustness analysis that holds a five-pipeline architecture matrix fixed and varies embedder (local e5-small vs. Azure text-embedding-3-small), corpus (DO-178C typed-edge requirements vs. Wikipedia paragraph chains via MuSiQue), and judge (paired GPT-5.4 x GPT-4.1 on both corpora), over 2x4,440 main-matrix runs, 600 cross-corpus runs, and over 5,000 faithfulness judgments. (C2a) GraphRAG's graph walk floods the context window at precision 0.12-0.23, but the synthesizer cites selectively at precision 0.48-0.65; scoring the retrieved set as the attribution set inverts the architecture ranking, which reconciles part of the disagreement in prior reports. (C1) Answer-level citation winners are corpus- and stratum-conditional but embedder-robust: GraphRAG ties vanilla on short-hop DO-178C queries and wins every MuSiQue stratum, while agentic pipelines lead only on 3+-hop requirements queries. (C2b) Faithfulness is corpus-conditional: on DO-178C it declines with hop distance (trend p<0.05 in three of four judge x embedder combinations); on Wikipedia chains neither judge shows a collapse. (C3) Single-judge LLM faithfulness is fragile to retrieval state: GPT-5.4's self-kappa across embedders is 0.137 (41% verdict change) against a same-day test-retest floor of 0.76, and re-judging frozen inputs eleven weeks later gives kappa <= 0.14 for both judges. A learned router on dense embeddings alone reaches macro-F1 0.86 on hop classification (C4). We argue that RAG architecture claims should be tested at this level of robustness -- including robustness to the citation-measurement point -- before they are trusted.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。