arXiv:2603.25944cs.CLcs.AI2026-03

小模型也能精准处理法律文本,关键在架构和训练质量。

Can Small Models Reason About Legal Documents? A Comparative Study

  • 用9个<10B参数模型测试三种法律任务,对比五种提示策略。
  • 3B参数的混合专家模型平均准确率媲美GPT-4o-mini,且更擅长识别法律结论。
  • 少样本提示最稳定,检索质量差异不大,成本仅62美元。

大语言模型在法律应用中表现优异,但部署前沿模型存在成本、延迟和隐私问题。我们评估了九个参数量低于100亿的模型在三个法律基准(ContractNLI、CaseHOLD、ECtHR)上的表现,采用五种提示策略(直接、思维链、少样本、BM25 RAG、密集检索RAG),共完成405次实验,每组配置随机种子三次。结果表明,仅激活30亿参数的混合专家模型在平均准确率上达到GPT-4o-mini水平,并在法律判决识别任务中表现更优;模型架构与训练质量比参数量更重要,最大模型(90亿参数)整体表现最差。思维链提示策略效果高度依赖任务——提升合同蕴含判断,却降低多选题法律推理表现;少样本提示成为最稳定的策略。对比BM25与密集检索在RAG中的表现,结果几乎相同,说明瓶颈在于语言模型对检索内容的利用能力,而非检索质量本身。所有实验通过云端推理接口完成,总成本仅62美元,证明无需专用GPU即可开展严谨的LLM评估。

原文摘要 · Abstract (English)

Large language models show promise for legal applications, but deploying frontier models raises concerns about cost, latency, and data privacy. We evaluate whether sub-10B parameter models can serve as practical alternatives by testing nine models across three legal benchmarks (ContractNLI, CaseHOLD, and ECtHR) using five prompting strategies (direct, chain-of-thought, few-shot, BM25 RAG, and dense RAG). Across 405 experiments with three random seeds per configuration, we find that a Mixture-of-Experts model activating only 3B parameters matches GPT-4o-mini in mean accuracy while surpassing it on legal holding identification, and that architecture and training quality matter more than raw parameter count. Our largest model (9B parameters) performs worst overall. Chain-of-thought prompting proves sharply task-dependent, improving contract entailment but degrading multiple-choice legal reasoning, while few-shot prompting emerges as the most consistently effective strategy. Comparing BM25 and dense retrieval for RAG, we find near-identical results, suggesting the bottleneck lies in the language model's utilization of retrieved context rather than retrieval quality. All experiments were conducted via cloud inference APIs at a total cost of $62, demonstrating that rigorous LLM evaluation is accessible without dedicated GPU infrastructure.

法律AI小模型RAG评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。