arXiv:2503.08379cs.IRcs.CL2025-03被引 6

首个巴西葡萄牙语法律信息检索数据集,支持关键词与问答查询评估。

JurisTCU: A Brazilian Portuguese Information Retrieval Dataset with Query Relevance Judgments

  • 构建16,045份巴西联邦审计法院判决文档,含150个带相关性标注的查询。
  • 基于短关键词查询时,文档扩展方法使BM25性能提升超45%,开放AI嵌入模型提升约70%。
  • 适合法律信息检索、多语言搜索及巴西本地化系统研究者使用。

本文介绍JurisTCU,一个面向巴西葡萄牙语法律信息检索(LIR)的公开数据集。该数据集包含来自巴西联邦审计法院的16,045份判例文档,以及150个经相关性标注的查询,涵盖真实用户关键词查询、合成关键词查询和合成问题查询三类。相关性判断采用大语言模型评分与领域专家验证相结合的混合方法生成。我们在14项实验中测试了词法检索(文档扩展方法)和语义检索(基于BERT和OpenAI嵌入)的效果。结果显示,文档扩展显著提升了标准BM25在短关键词查询上的表现,各项指标(P@10、R@10、nDCG@10)提升超过45%;而OpenAI嵌入模型表现最优,相同条件下指标提升约70%,表明其在捕捉该领域语义关系方面优于依赖词法匹配的方法。该数据集为葡萄牙语信息检索研究提供基础,同时有助于提升对巴西公民具有重要意义的法律搜索系统性能。

原文摘要 · Abstract (English)

This paper introduces JurisTCU, a Brazilian Portuguese dataset for legal information retrieval (LIR). The dataset is freely available and consists of 16,045 jurisprudential documents from the Brazilian Federal Court of Accounts, along with 150 queries annotated with relevance judgments. It addresses the scarcity of Portuguese-language LIR datasets with query relevance annotations. The queries are organized into three groups: real user keyword-based queries, synthetic keyword-based queries, and synthetic question-based queries. Relevance judgments were produced through a hybrid approach combining LLM-based scoring with expert domain validation. We used JurisTCU in 14 experiments using lexical search (document expansion methods) and semantic search (BERT-based and OpenAI embeddings). We show that the document expansion methods significantly improve the performance of standard BM25 search on this dataset, with improvements exceeding 45% in P@10, R@10, and nDCG@10 metrics when evaluating short keyword-based queries. Among the embedding models, the OpenAI models produced the best results, with improvements of approximately 70% in P@10, R@10, and nDCG@10 metrics for short keyword-based queries, suggesting that these dense embeddings capture semantic relationships in this domain, surpassing the reliance on lexical terms. Besides offering a dataset for the Portuguese-language IR research community, suitable for evaluating search systems, the results also contribute to enhancing a search system highly relevant to Brazilian citizens.

法律检索葡萄牙语信息检索数据集

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。