构建首个大规模澳洲法律引文预测基准,验证各类大模型方法效果。
Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study
- 构建5.5万条澳洲法律实例数据集,评估提示、微调与检索增强等多种方法。
- 指令微调使通用开源模型表现最佳,混合方法结合重排序更优。
- 适合法律AI研究者,尤其关注法律文本理解与引文生成的团队。
大型语言模型在法律任务中展现强大潜力,但法律引文预测仍研究不足。该任务需精细上下文理解与精准法规/判例识别。本文提出AusLaw引文基准,包含5.5万条澳洲法律实例和18,677个唯一引文,据知为同类中规模最大。系统评估四类方案:(i) 通用与法律专用LLM的直接提示,(ii) 使用通用或领域嵌入的纯检索管道,(iii) 监督微调,(iv) 结合查询扩展、投票集成或重排序的混合策略。结果表明,通用与法律专用模型作为独立方案表现极差,接近零分。在特定数据集上进行指令微调(甚至使用通用开源模型)表现最优。数据库粒度与嵌入类型对检索方法影响显著,采用训练好的重排序器的混合方法表现最佳。尽管如此,性能仍存在近50%差距,凸显该基准在法律领域研究中的严格测试价值。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated strong potential across legal tasks, yet the problem of legal citation prediction remains under-explored. At its core, this task demands fine-grained contextual understanding and precise identification of relevant legislation or precedent. We introduce the AusLaw Citation Benchmark, a real-world dataset comprising 55k Australian legal instances and 18,677 unique citations which to the best of our knowledge is the first of its scale and scope. We then conduct a systematic benchmarking across a range of solutions: (i) standard prompting of both general and law-specialised LLMs, (ii) retrieval-only pipelines with both generic and domain-specific embeddings, (iii) supervised fine-tuning, and (iv) several hybrid strategies that combine LLMs with retrieval augmentation through query expansion, voting ensembles, or re-ranking. Results show that neither general nor law-specific LLMs suffice as stand-alone solutions, with performance near zero. Instruction tuning (of even a generic open-source LLM) on task-specific dataset is among the best performing solutions. We highlight that database granularity along with the type of embeddings play a critical role in retrieval-based approaches, with hybrid methods which utilise a trained re-ranker delivering the best results. Despite this, a performance gap of nearly 50% remains, underscoring the value of this challenging benchmark as a rigorous test-bed for future research in legal-domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。