arXiv:2603.23857cs.AIcs.CY2026-03

AI法律助手会伪造真实存在的判例,且这种错误可预测。

When AI output tips to bad but nobody notices: Legal implications of AI's mistakes

  • 基于变压器机制分析,发现AI输出错误有确定性阈值
  • 模拟起草文件中,伪造内容具高度可信度且无法察觉
  • 建议律师与法院建立验证流程,应对技术设计缺陷

生成式AI在商业与法律领域的应用带来显著效率提升,但尤其在法律领域,其存在一种危险的失效模式:AI会编造看似真实的判例、法规和司法裁决。律师若无意提交此类虚假内容,将面临职业处罚、过失责任及声誉损失,法院则面临对抗制程序完整性受威胁。该现象常被归为随机‘幻觉’,但近期对Transformer核心机制的物理分析揭示其具有可计算的确定性成分——当AI内部状态跨越特定阈值时,输出会从可靠法律推理突变为权威性伪造。本文在法律行业场景下展示该科学原理,通过模拟文书起草过程说明,伪造风险并非偶发故障,而是技术设计的可预见后果,直接关联日益重要的技术胜任义务。我们建议法律从业者、法院与监管机构摒弃‘黑箱’思维,采用基于系统实际失效机制的验证协议。

原文摘要 · Abstract (English)

The adoption of generative AI across commercial and legal professions offers dramatic efficiency gains -- yet for law in particular, it introduces a perilous failure mode in which the AI fabricates fictitious case law, statutes, and judicial holdings that appear entirely authentic. Attorneys who unknowingly file such fabrications face professional sanctions, malpractice exposure, and reputational harm, while courts confront a novel threat to the integrity of the adversarial process. This failure mode is commonly dismissed as random `hallucination', but recent physics-based analysis of the Transformer's core mechanism reveals a deterministic component: the AI's internal state can cross a calculable threshold, causing its output to flip from reliable legal reasoning to authoritative-sounding fabrication. Here we present this science in a legal-industry setting, walking through a simulated brief-drafting scenario. Our analysis suggests that fabrication risk is not an anomalous glitch but a foreseeable consequence of the technology's design, with direct implications for the evolving duty of technological competence. We propose that legal professionals, courts, and regulators replace the outdated `black box' mental model with verification protocols based on how these systems actually fail.

AI误判法律AI伪造判例技术问责

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。