对比两种论文问答方法:向量检索与大模型编写的维基,结果因评价标准而异。
Single-Round Vector RAG vs an LLM-Compiled Wiki: A Preregistered Comparison on a Small Multi-Domain Research Corpus
- 用单轮向量检索和大模型生成维基两种方式回答问题
- 维基在跨论文整合上表现更好,但查询耗能高出21倍
- 不同评分方式导致结果分歧,无单一最优方案
我们预先注册了一项对比实验,评估两种帮助大模型回答小规模多领域研究语料中问题的方法:单轮向量检索(Vector RAG)系统与由大模型编写的Markdown维基,由工具型代理浏览。两者使用相同答案生成模型,回答了13个问题,覆盖24篇论文,由两名盲评大模型判分。三个预注册假设中,一个弱支持、一个支持、一个被推翻。维基预测在跨论文综合上更优,其连接发现能力显著更强,但合并两评委分数后组织优势未达注册阈值。RAG预测在单事实查询上表现稳定,符合标准,但第二名评委单独判断会否定该结论。维基预测构建成本高、查询成本低;构建成本差距约两个数量级,但查询成本反转:维基每查询耗时21倍于RAG,无成本平衡点。两项探索分析解释分歧:分解检索型RAG以更低代价几乎消除维基的综合优势,但仍保留引文支持优势。整体依据评分与原子化引文检查方向相反,评委间一致性极低(rho=0.04),而具体指标一致性高(rho=0.81)。因此,基于证据的研究合成并非单一能力,各系统在证据组织、引文支持程度和运行成本上各有优劣,无一在三方面全胜。谁胜出取决于检索基线、评分粒度和评判者。
原文摘要 · Abstract (English)
We preregistered a comparison of two ways to help an LLM answer questions over a small research corpus: a single-round Vector RAG system and an LLM-compiled markdown wiki browsed by a tool-using agent. Both systems answered the same 13 questions over 24 papers using the same answer-generating model, and their answers were scored by two blinded LLM judges. The three preregistered predictions, in registered order, came out one weakly supported, one supported, and one refuted. The wiki was predicted to synthesize better across papers; it scored much better at connecting findings, but its organization advantage fell below the registered threshold once both judges' scores were combined. RAG was predicted to hold its own on single-fact lookup, and it met the registered test, though the second judge alone would have refuted it. The wiki was predicted to be expensive to build and cheap to query; the build side held by roughly two orders of magnitude, but the query side reversed: the wiki spent about 21 times more tokens per query, so no break-even point exists. Two exploratory analyses explain the disagreement. A decomposition-retrieval variant of RAG removes almost all of the wiki's synthesis advantage at lower token cost, though not its advantage in claim-by-claim citation support. Holistic groundedness scoring disagrees with atomized citation checking by direction, and between judges: rank agreement on that criterion is near zero (rho = 0.04), against rho = 0.81 on the most concretely defined criterion. Grounded research synthesis is therefore not a single capability: systems differ in how well they organize evidence, how well their citations support each claim, and what they cost to run, and no architecture here was best on all three. Which one appears to win depends on the retrieval baseline, the scoring granularity, and the judge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。