大模型未必懂普通人如何用语言，法律解释中不可轻信。
Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysis
- 用控制实验检验大模型对普通语言理解的判断力
- 微小提问格式变化导致结论大幅波动，可靠性差
- 模型判断与人类意见相关性弱，不适合高风险法律场景
在美国司法体系中，法律解释常依赖于文本对‘普通人’语言使用的理解。近期有研究建议法律从业者使用大语言模型（LLMs）来判断文本的普通含义。但这些模型是否胜任？本文通过控制实验表明，当前大模型在普通语言理解任务上存在显著脆弱性：仅改变问题表述格式，模型输出即可能产生巨大差异，这种可操纵性可能被利益相关方利用。与人类对类似法律解释问题的回答数据集对比发现，模型判断与人类意见仅有中等程度相关，不足以支撑高风险法律决策。因此，单纯依赖大规模预训练无法确保模型具备处理日常语言理解任务的能力，需谨慎评估其在法律实践中的应用。
原文摘要 · Abstract (English)
In the U.S. judicial system, a widespread approach to legal interpretation entails assessing how a legal text would be understood by an `ordinary' speaker of the language. Recent scholarship has proposed that legal practitioners leverage large language models (LLMs) to ascertain a text's ordinary meaning. But are LLMs up to the task? As textual interpretation questions arise in spheres ranging from criminal law to civil rights, we argue it is crucial that models not be taken as authoritative without rigorous evaluation. This work offers an empirical argument against LLM-assisted interpretation as recently practiced by legal scholars and federal judges, who reasoned the large amount of data that models see in training would enable models to illuminate how people ordinarily use certain words or phrases. In controlled experiments, we find failures in robustness which cast doubt on this assumption and raise serious questions about the utility of these models in practice. For the models in our evaluation, slight changes to the format of a question can lead to wildly different conclusions -- a vulnerability that parties with an interest in the outcome could exploit. Comparing with a dataset where people were asked similar legal interpretation questions, we see that these models are at best moderately correlated to human judgments -- not strong enough given the stakes in this domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。