arXiv:2603.01710cs.CLcs.IR2026-03

构建首个法律RAG端到端评测基准,揭示检索环节决定系统性能上限。

Legal RAG Bench: an end-to-end benchmark for legal RAG

  • 设计包含4876段法律文本和100个复杂问题的评测集
  • 发现检索模型贡献度远超大模型,提升检索准确率34点
  • 揭示多数‘幻觉’实为检索失败所致,适合法律AI研究者参考

我们提出Legal RAG Bench,一个用于评估法律RAG系统端到端性能的基准与评测方法。该基准包含来自维多利亚刑事指控书的4,876段文本,以及100个需刑事法与程序知识的复杂人工构造问题,提供长答案和支撑文本。评测方法采用全因子设计与新型分层错误分解框架,实现对检索与推理模型贡献的公平比较。我们评估了三种主流嵌入模型(Isaacus' Kanon 2 Embedder、Google Gemini Embedding 001、OpenAI Text Embedding 3 Large)和两种前沿大模型(Gemini 3.1 Pro、GPT-5.2),发现信息检索是法律RAG性能的主要驱动力,大模型对正确性与一致性影响较弱。Kanon 2 Embedder表现最佳,平均正确率提升17.5点,一致性提升4.5点,检索准确率提升34点。我们观察到许多被归因于‘幻觉’的错误实由检索失败引发,结论是检索环节设定了现代法律RAG系统的性能上限。本文详述构建过程与评估结果,并开源代码与数据以支持复现。

原文摘要 · Abstract (English)

We introduce Legal RAG Bench, a benchmark and evaluation methodology for assessing the end-to-end performance of legal RAG systems. As a benchmark, Legal RAG Bench consists of 4,876 passages from the Victorian Criminal Charge Book alongside 100 complex, hand-crafted questions demanding expert knowledge of criminal law and procedure. Both long-form answers and supporting passages are provided. As an evaluation methodology, Legal RAG Bench leverages a full factorial design and novel hierarchical error decomposition framework, enabling apples-to-apples comparisons of the contributions of retrieval and reasoning models in RAG. We evaluate three state-of-the-art embedding models (Isaacus' Kanon 2 Embedder, Google's Gemini Embedding 001, and OpenAI's Text Embedding 3 Large) and two frontier LLMs (Gemini 3.1 Pro and GPT-5.2), finding that information retrieval is the primary driver of legal RAG performance, with LLMs exerting a more moderate effect on correctness and groundedness. Kanon 2 Embedder, in particular, had the largest positive impact on performance, improving average correctness by 17.5 points, groundedness by 4.5 points, and retrieval accuracy by 34 points. We observe that many errors attributed to hallucinations in legal RAG systems are in fact triggered by retrieval failures, concluding that retrieval sets the ceiling for the performance of many modern legal RAG systems. We document why and how we built Legal RAG Bench alongside the results of our evaluations. We also openly release our code and data to assist with reproduction of our findings.

法律AIRAG评测检索增强大模型评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。