首个面向法律咨询对话的RAG评估基准，助力大模型精准回应法律问题。
LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversation
- 构建多轮法律咨询对话数据集，每轮提问逐步深入。
- 测试结果显示现有RAG系统在法律上下文理解上仍有明显短板。
- 适合法律AI研究者、司法科技公司及合规工具开发者参考。
检索增强生成（RAG）在多个领域显著提升了大语言模型（LLMs）的表现，但在法律领域尚无专用评估基准，制约了该方向的发展。为此，我们提出LexRAG，首个用于评估多轮法律咨询对话中RAG系统的基准。LexRAG包含1,013组多轮对话样本和17,228篇候选法律条文，每组对话由法律专家标注，涵盖五轮渐进式提问。任务包括：（1）对话式知识检索，需基于多轮上下文准确召回相关法律条文；（2）回答生成，要求生成合法且合理的答复。为保障可复现性，我们开发了LexiT法律RAG工具包，提供针对法律场景的完整RAG组件实现。同时引入基于LLM的评估流水线，实现高效细致的评测。通过多种LLMs与检索方法的实验分析，揭示了当前RAG系统在处理法律对话中的关键局限。LexRAG为法律领域RAG应用提供了新基准，代码与数据已开源。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) has proven highly effective in improving large language models (LLMs) across various domains. However, there is no benchmark specifically designed to assess the effectiveness of RAG in the legal domain, which restricts progress in this area. To fill this gap, we propose LexRAG, the first benchmark to evaluate RAG systems for multi-turn legal consultations. LexRAG consists of 1,013 multi-turn dialogue samples and 17,228 candidate legal articles. Each sample is annotated by legal experts and consists of five rounds of progressive questioning. LexRAG includes two key tasks: (1) Conversational knowledge retrieval, requiring accurate retrieval of relevant legal articles based on multi-turn context. (2) Response generation, focusing on producing legally sound answers. To ensure reliable reproducibility, we develop LexiT, a legal RAG toolkit that provides a comprehensive implementation of RAG system components tailored for the legal domain. Additionally, we introduce an LLM-as-a-judge evaluation pipeline to enable detailed and effective assessment. Through experimental analysis of various LLMs and retrieval methods, we reveal the key limitations of existing RAG systems in handling legal consultation conversations. LexRAG establishes a new benchmark for the practical application of RAG systems in the legal domain, with its code and data available at https://github.com/CSHaitao/LexRAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。