混合检索提升问答准确率，动态测试验证系统性能。
Evaluating Hybrid Retrieval Augmented Generation using Dynamic Test Sets: LiveRAG Challenge
- 融合稀疏（BM25）与稠密（E5）检索，结合大模型生成答案。
- 神经重排序使MAP提升52%，但耗时增加至84秒/题。
- 词汇匹配度是表现关键，适合关注生成可信性的研究者。
我们提交了参加2025年LiveRAG挑战的方案，评估检索增强生成（RAG）系统在动态测试集上的表现，使用FineWeb-10BT语料库。最终采用混合方法，结合稀疏（BM25）与稠密（E5）检索，并用Falcon3-10B-Instruct生成相关且可信的答案。在200个由DataMorgana生成的合成问题、64种用户组合的系统评估中，神经重排序（RankLLaMA）使MAP从0.523提升至0.797（相对提升52%），但每题耗时增至84秒（原为1.74秒）。虽然DSPy优化提示策略在语义相似度上达到0.771（高于基线0.668），但其0%拒绝率引发对过度自信和泛化能力的担忧。未使用重排序的混合系统在25支队伍中获得忠实性第4名、正确性第11名。分析显示，问题与文档间词汇对齐程度是性能最强预测因子，文档表述相似可使余弦相似度从0.562升至0.762。
原文摘要 · Abstract (English)
We present our submission to the LiveRAG Challenge 2025, which evaluates retrieval-augmented generation (RAG) systems on dynamic test sets using the FineWeb-10BT corpus. Our final hybrid approach combines sparse (BM25) and dense (E5) retrieval methods and then aims to generate relevant and faithful answers with Falcon3-10B-Instruct. Through systematic evaluation on 200 synthetic questions generated with DataMorgana across 64 unique question-user combinations, we demonstrate that neural re-ranking with RankLLaMA improves MAP from 0.523 to 0.797 (52% relative improvement) but introduces prohibitive computational costs (84s vs 1.74s per question). While DSPy-optimized prompting strategies achieved higher semantic similarity (0.771 vs 0.668), their 0% refusal rates raised concerns about over-confidence and generalizability. Our submitted hybrid system without re-ranking achieved 4th place in faithfulness and 11th place in correctness among 25 teams. Analysis across question categories reveals that vocabulary alignment between questions and documents was the strongest predictor of performance on our development set, with document-similar phrasing improving cosine similarity from 0.562 to 0.762.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。