为土耳其法律领域打造专用大模型，提升法律文本理解能力。
HUKUKBERT: Domain-Specific Language Model for Turkish Law
- 基于19GB清洗语料，融合多种掩码策略训练
- 法律填空测试达84.4%准确率，文书结构分割通过率达92.8%
- 适合法律文本识别、判决预测等场景使用
自然语言处理进展推动了法律科技系统的发展，但土耳其法律领域仍缺乏专用数据与模型。英文有LEGAL-BERT等法律编码器，而土耳其尚无类似高体量模型。我们提出HukukBERT，一个在19GB清洗语料上训练的土耳其法律语言模型，采用混合领域自适应预训练（DAPT）方法，结合全词掩码、词段掩码、词元段掩码和关键词定向掩码。对比通用及现有土耳其领域模型，其48K WordPiece分词器与DAPT流程表现更优。在法律填空测试（针对土耳其法院判决的掩码术语预测基准）中，HukukBERT达到84.40%的Top-1准确率，且测试文本未出现在预训练语料中，避免了数据泄露。在官方土耳其法院判决的结构化分割下游任务中，文档通过率达92.8%。HukukBERT已开源，可用于命名实体识别、判决预测与文档分类等任务。
原文摘要 · Abstract (English)
Natural language processing (NLP) advances have powered a generation of LegalTech systems, but Turkish law remains under-served by domain-specific data and models. English has legal encoders such as LEGAL-BERT; no comparable high-volume Turkish counterpart exists. We introduce HukukBERT, a Turkish legal language model trained on a 19 GB cleaned corpus using a hybrid domain-adaptive pre-training (DAPT) recipe that mixes Whole-Word Masking, Token Span Masking, Word Span Masking, and targeted Keyword Masking. We compared our 48K WordPiece tokenizer and DAPT pipeline against general-purpose and existing domain-specific Turkish models. On the Legal Cloze Test - a masked legal term prediction benchmark over Turkish court decisions - HukukBERT reaches 84.40% Top-1 accuracy and beats every baseline we tested. The Legal Cloze Test is synthetically constructed, so its passages are absent from the pre-training corpus by construction, eliminating train-test contamination. On the downstream task of structural segmentation of official Turkish court decisions, it reaches a 92.8% document pass rate. We release HukukBERT to support Turkish legal NLP work in named entity recognition, judgment prediction, and document classification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。