大模型常错用最新法律，反而越聪明越容易出错。
When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning
- 发现大模型有强倾向用最新法律，无视案件发生时间。
- 越擅长通用推理的模型，在时间法律判断上表现越差。
- 原因可能是强化学习让推理路径变单一，只选当前法律。
法律推理任务如法律判决预测（LJP）需要识别与案件时间相匹配的法律版本——我们称之为时间适用法判定。然而，大语言模型（LLMs）是否能可靠完成此任务仍未知。本文构建了一个基准测试，系统评估LLMs在时间适用法判定上的表现，并深入探究其失败原因。实验揭示四个关键发现：第一，LLMs表现出对最新颁布法律的强烈偏好，无论案件事实发生在何时；第二，这种偏差并非源于无法理解法律的时间范围，也非缺乏历史法律知识；第三，行为证据表明，强化学习塑造的显式推理可能是关键机制：尽管提升了通用推理能力，却降低了推理路径的多样性，导致模型趋于应用当前法律；第四，这产生了反直觉的逆相关关系：通用推理能力越强的模型，在时间法律推理上表现反而越差。研究为未来提升大模型在时间约束法律推理中的性能提供了具体指导。
原文摘要 · Abstract (English)
Legal reasoning tasks such as legal judgment prediction (LJP) require identifying the temporally correct version of the law governing a case -- a capability we term temporal applicable-law determination. However, whether large language models (LLMs) can reliably perform this task remains unexplored. In this paper, we construct a benchmark to evaluate LLMs on temporal applicable-law determination, and systematically investigate why they fail at temporal legal reasoning. Our experiments reveal four key findings. First, LLMs exhibit a strong bias toward applying the most recently enacted law, regardless of when the legally relevant facts occurred. Second, this bias does not stem from an inability to understand that laws have temporal scope, nor from a lack of knowledge about historical statutes. Third, we provide behavioral evidence that reinforcement-learning-shaped explicit reasoning may be a key mechanism: while improving general reasoning ability, it reduces the diversity of reasoning paths, causing models to converge on applying the current law. Fourth, this produces a counterintuitive inverse relationship: models with stronger general reasoning ability tend to perform worse on temporal legal reasoning. Our findings offer concrete guidance for future work on improving LLM performance in temporally grounded legal reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。