法律AI常错还自信,学生也难辨真伪,需新教学与验证机制。
Can Legal AI Know When It Is Wrong? And Do Students Know When It Is?
- 用高置信度错误率量化模型误判,发现其对法律更新反应迟钝。
- Meta AI错误率最高达31.7%,且平均信心值9.1/10,风险极重。
- 学生普遍缺乏AI伦理培训,易受幻觉误导,需强化验证能力。
将大语言模型(LLMs)引入印度司法体系虽可提升司法可及性,但带来严重风险。我们识别出‘信心惯性’现象——即模型在错误判决时仍表现出近乎最大自信,类似达克效应,源于假设的‘判例过拟合’偏差。第一阶段社会技术审计测试了ChatGPT(GPT-5.2)、Meta AI和Perplexity AI在60个涉及《1872年印度合同法》及特定履行法定执行转变案例上的表现。我们提出高置信度错误率(HCER)来衡量以危险确定性(≥9分,满分10分)输出的错误判决。所有模型在法律更新上表现不佳。Meta AI最脆弱(HCER 31.7%),频繁误用修订前规则,平均信心9.1/10;其次为Perplexity(15.0%)和ChatGPT(6.7%)。第二阶段通过问卷调查380名印度法学生,研究人类对这种过度自信的脆弱性。验证行为多为被动应对机器幻觉:遭遇虚构引用的学生验证评分(4.2/5)显著高于未遇者(2.8/5)。尽管81.6%的学生知道提交幻觉案例可能导致藐视法庭,但71.1%从未接受过正式的伦理AI使用培训。我们建议转向对抗性法律研究教学法,并实施基于来源的验证架构,防止系统性职业失职。
原文摘要 · Abstract (English)
Integrating Large Language Models (LLMs) into the Indian judiciary promises access to justice but introduces severe risks. We identify the 'inertia of confidence'--an overconfidence phenomenon analogous to the Dunning-Kruger effect where LLMs provide incorrect legal verdicts with near-maximum confidence, driven by a hypothesized 'precedent overfitting' bias. Phase I of our socio-technical audit tested ChatGPT (GPT-5.2), Meta AI, and Perplexity AI on a 60-case battery regarding the Indian Contract Act, 1872, and the shift toward statutory enforcement of specific performance. We introduce the High-Confidence Error Rate (HCER) to quantify incorrect verdicts delivered with dangerous certainty (>= 9 on a 1-10 scale). All models struggled with statutory updates. Meta AI proved most vulnerable (31.7% HCER), frequently misapplying pre-amendment rules with a 9.1/10 mean confidence, followed by Perplexity (15.0%) and ChatGPT (6.7%). Phase II investigated human vulnerability to this overconfidence via a survey of Indian law students (N=380). Verification often functions as a reactive adaptation to machine hallucinations: students encountering fabricated citations reported higher verification scores (4.2/5) than those with no such encounters (2.8/5). Furthermore, while 81.6% knew submitting hallucinated cases can lead to contempt-of-court, 71.1% received no formal training on ethical AI use. We propose shifting toward adversarial legal research pedagogy and implementing source-grounded verification architectures to prevent systemic professional negligence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。