法律预训练与微调对大模型法律概念理解能力影响不一
Impacts of Continued Legal Pre-Training and IFT on LLMs' Latent Representations of Human-Defined Legal Concepts
- 对比三模型在法律文本上的注意力分布变化
- 部分法律概念的注意力提升不明显，整体结构不匹配人类认知
- 适合关注法律AI可解释性与模型训练策略的研究者
本文旨在为人工智能与法律领域的研究者及从业者提供更深入的理解：在法律语料上进行持续预训练和指令微调（IFT）是否能增强大语言模型（LLMs）在生成输入序列全局上下文表征时对人类定义的法律概念的利用。我们比较了三个模型：Mistral 7B、SaulLM-7B-Base（在法律语料上持续预训练的Mistral 7B）和SaulLM-7B-Instruct（进一步进行IFT）。通过分析来自近期AI&Law文献的7个包含人类定义法律概念的文本片段，我们首先比较了各模型对代表法律概念的词元子集分配的总注意力比例；随后可视化原始注意力得分的变化模式，评估法律训练是否引入了对应于人类法律知识结构的新注意力模式。结果表明：(1) 法律训练的影响在不同人类定义的法律概念间分布不均；(2) 法律训练中学习到的法律知识上下文表征，并未与人类定义的法律概念结构一致。最后提出未来研究方向，探讨法律大模型训练的动态机制。
原文摘要 · Abstract (English)
This paper aims to offer AI & Law researchers and practitioners a more detailed understanding of whether and how continued pre-training and instruction fine-tuning (IFT) of large language models (LLMs) on legal corpora increases their utilization of human-defined legal concepts when developing global contextual representations of input sequences. We compared three models: Mistral 7B, SaulLM-7B-Base (Mistral 7B with continued pre-training on legal corpora), and SaulLM-7B-Instruct (with further IFT). This preliminary assessment examined 7 distinct text sequences from recent AI & Law literature, each containing a human-defined legal concept. We first compared the proportions of total attention the models allocated to subsets of tokens representing the legal concepts. We then visualized patterns of raw attention score alterations, evaluating whether legal training introduced novel attention patterns corresponding to structures of human legal knowledge. This inquiry revealed that (1) the impact of legal training was unevenly distributed across the various human-defined legal concepts, and (2) the contextual representations of legal knowledge learned during legal training did not coincide with structures of human-defined legal concepts. We conclude with suggestions for further investigation into the dynamics of legal LLM training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。