对比七款模型在乌克兰法律文本上的分词效率与零样本表现,发现分词成本差异达1.6倍。
Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Study
- 比较不同模型对乌克兰法律文本的分词效率,揭示分词成本差异显著
- NVIDIA Nemotron Super 3以83.1分最高,参数多5.6倍但成本仅为三分之一
- 零样本优于少样本,且战时法律文本难以被旧模型理解
在乌克兰国家法院数据库(EDRSR)的273份经验证判决上,我们对来自五个厂商的七款基础模型进行了基准测试,评估其分词效率与零样本性能。结果显示:(1)Qwen 3模型比Llama系列多消耗60%的令牌,分词分析是成本优化部署的前提;(2)NVIDIA Nemotron Super 3(120B)获得最高综合得分83.1,优于参数多5.6倍的Mistral Large 3,且仅需其三分之一的API成本,表明模型规模无法代表领域表现;(3)少样本提示导致性能下降高达26个百分点,且该现象源于乌克兰语表达特性,非示例选择偏差;(4)基于战前判决(2008–2013)训练的分类器在战时判决(2022–2026)上性能下降27.9个百分点,且存在明显前后不对称性:新模型可向后迁移(提升14.6个百分点),旧模型则在战时语言中严重失效。研究建议:模型选型前应优先分析分词效率,零样本优于少样本。为支持可复现性并填补乌克兰法律语料空白,我们公开发布包含14,452份判决的语料库,涵盖2008–2026年、三个时间阶段及七类判决标签。
原文摘要 · Abstract (English)
Tokenizer fertility varies 1.6x across foundation models on Ukrainian legal text, yet this cost-critical dimension is absent from model selection practice. We benchmark seven models from five providers on 273 validated court decisions from Ukraine's state registry (EDRSR), measuring tokenizer fertility and zero-shot performance on three tasks. Four findings emerge. (1) Qwen 3 models consume 60% more tokens than Llama-family models on identical input, making tokenizer analysis a prerequisite for cost-efficient deployment. (2) NVIDIA Nemotron Super 3 (120B) achieves the highest composite score (83.1), outperforming Mistral Large 3 (5.6x more total parameters) at one-third the API cost model scale is a poor proxy for domain performance. (3) Few-shot prompting degrades performance by up to 26 percentage points; stratified and prompt-sensitivity ablations confirm this is intrinsic to Ukrainian-language demonstrations, not an artifact of example selection. (4) A cross-temporal generalization experiment reveals that classifiers trained on pre-war court ecisions (2008-2013) lose 27.9 percentage points when applied to full-scale invasion era decisions (2022-2026), with a pronounced forward-backward asymmetry: newer models transfer backward (+14.6 pp above forward transfer), but older models fail catastrophically on wartime legal language. For practitioners: tokenizer analysis should precede model selection, and zero-shot is a more reliable default than few-shot for morphologically rich languages. To support reproducibility and address the absence of Ukrainian from legal NLP benchmarks, we release a public dataset of 14,452 court decisions spanning 2008-2026, annotated with seven outcome labels across three temporal epochs that capture the impact of armed conflict on judicial proceedings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。