分析大模型在法律文本中的归因差异，揭示其判断依据
Attribution analysis of legal language as used by LLM
- 用积分梯度法分析法律大模型与通用BERT的决策依据
- 不同模型对同一案件判断结果不一致，归因可定位原因
- 法律术语和分词差异是模型表现差异主因，适合法律AI研究者
针对三个专为法律任务设计的公开大语言模型，我们发现基于法律语料训练可提升分类准确率，但其背后机制尚不明确。本文使用两个公开法律数据集：一个简单的二分类任务（判断是否推翻判例），以及更复杂的多选任务（识别司法判决中的裁判要旨）。通过对比法律大模型与通用BERT模型在两组数据上的表现，采用积分梯度归因技术，分析模型预测差异的根源，并从分词方式角度进行解释。实验表明，尽管所有模型均能正确分类部分案例，但某些案例仅被单一模型正确识别，归因分析可有效揭示原因。研究发现，模型分词器差异是导致行为不同的主要原因。结合数据集中生成词汇的频率分析与已知停用词表，可识别出显著代表法律主题的关键词元。
原文摘要 · Abstract (English)
Three publicly-available LLM specifically designed for legal tasks have been implemented and shown that classification accuracy can benefit from training over legal corpora, but why and how? Here we use two publicly-available legal datasets, a simpler binary classification task of ``overruling'' texts, and a more elaborate multiple choice task identifying ``holding'' judicial decisions. We report on experiments contrasting the legal LLM and a generic BERT model for comparison, against both datasets. We use integrated gradient attribution techniques to impute ``causes'' of variation in the models' perfomance, and characterize them in terms of the tokenizations each use. We find that while all models can correctly classify some test examples from the casehold task, other examples can only be identified by only one, model, and attribution can be used to highlight the reasons for this. We find that differential behavior of the models' tokenizers accounts for most of the difference and analyze these differences in terms of the legal language they process. Frequency analysis of tokens generated by dataset texts, combined with use of known ``stop word'' lists, allow identification of tokens that are clear signifiers of legal topics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。