专为法律领域RAG系统设计的全流程评估工具。
LRAGE: Legal Retrieval Augmented Generation Evaluation Tool
- 提供图形与命令行界面，支持五类组件协同测试
- 多语言法律数据集验证，准确率随组件变化可追踪
- 开源工具，适合法律AI研究者快速实验迭代
近年来，构建检索增强生成（RAG）系统以提升大语言模型（LLM）能力已成为普遍做法。尤其在法律领域，遵循判例法（stare decisis）原则，先前判决对当前裁决具有重要意义。然而，RAG系统的整体性能依赖于五个关键组件：(1) 检索语料库、(2) 检索算法、(3) 重排序器、(4) LLM主干模型、(5) 评估指标。为此，我们提出LRAGE——一个面向法律领域的开源端到端评估工具。该工具提供图形界面（GUI）和命令行接口（CLI），便于开展系统性实验，分析上述五类组件变化对整体准确率的影响。我们在多语言法律基准上进行了验证，包括韩语（KBL）、英语（LegalBench）和中文（LawBench），展示了不同组件组合下的性能变化。代码已开源：https://github.com/hoorangyee/LRAGE。
原文摘要 · Abstract (English)
Recently, building retrieval-augmented generation (RAG) systems to enhance the capability of large language models (LLMs) has become a common practice. Especially in the legal domain, previous judicial decisions play a significant role under the doctrine of stare decisis which emphasizes the importance of making decisions based on (retrieved) prior documents. However, the overall performance of RAG system depends on many components: (1) retrieval corpora, (2) retrieval algorithms, (3) rerankers, (4) LLM backbones, and (5) evaluation metrics. Here we propose LRAGE, an open-source tool for holistic evaluation of RAG systems focusing on the legal domain. LRAGE provides GUI and CLI interfaces to facilitate seamless experiments and investigate how changes in the aforementioned five components affect the overall accuracy. We validated LRAGE using multilingual legal benches including Korean (KBL), English (LegalBench), and Chinese (LawBench) by demonstrating how the overall accuracy changes when varying the five components mentioned above. The source code is available at https://github.com/hoorangyee/LRAGE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。