首个巴西法律文书分类基准，验证微调模型在法律领域显著优于通用大模型。
LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification
- 用启发式校验的LLM辅助标注构建3105条巴西上诉案数据集。
- 微调后的BERTimbau-LoRA准确率87.6%，行政法类F1达0.91，远超商用模型。
- 适合关注葡萄牙语法律NLP、模型偏见与领域适配的研究者。
我们提出LegalBench-BR，首个面向巴西法律文本分类的公开评估基准。数据集包含3,105个来自圣卡塔琳娜州法院（TJSC）的上诉案件，通过DataJud API（CNJ）获取，并经由LLM辅助标注与启发式验证，覆盖五个法律领域。在类别平衡的测试集上，仅更新0.3%参数的BERTimbau-LoRA模型达到87.6%准确率和0.87宏平均F1，较Claude 3.5 Haiku提升22个百分点，较GPT-4o mini提升28个百分点。在行政法类别上，GPT-4o mini F1为0.00，Claude 3.5 Haiku为0.08，而该模型达到0.91。两类商用模型均系统性偏向民事法，无法区分模糊类别，而领域微调可消除此缺陷。结果表明，即使面对简单的五分类任务，通用大模型也无法替代领域适配模型，且在消费级显卡上通过LoRA微调可零成本实现性能跃升。我们开放完整数据集、模型与流程，支持葡萄牙语法律NLP的可复现研究。
原文摘要 · Abstract (English)
We introduce LegalBench-BR, the first public benchmark for evaluating language models on Brazilian legal text classification. The dataset comprises 3,105 appellate proceedings from the Santa Catarina State Court (TJSC), collected via the DataJud API (CNJ) and annotated across five legal areas through LLM-assisted labeling with heuristic validation. On a class-balanced test set, BERTimbau-LoRA, updating only 0.3% of model parameters, achieves 87.6% accuracy and 0.87 macro-F1 (+22pp over Claude 3.5 Haiku, +28pp over GPT-4o mini). The gap is most striking on administrativo (administrative law): GPT-4o mini scores F1 = 0.00 and Claude 3.5 Haiku scores F1 = 0.08 on this class, while the fine-tuned model reaches F1 = 0.91. Both commercial LLMs exhibit a systematic bias toward civel (civil law), absorbing ambiguous classes rather than discriminating them, a failure mode that domain-adapted fine-tuning eliminates. These results demonstrate that general-purpose LLMs cannot substitute for domain-adapted models in Brazilian legal classification, even when the task is a simple 5-class problem, and that LoRA fine-tuning on a consumer GPU closes the gap at zero marginal inference cost. We release the full dataset, model, and pipeline to enable reproducible research in Portuguese legal NLP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。