arXiv:2604.26233cs.AIcs.CY2026-04中稿 · the Proceedings of…

测试大模型在法律辩论中被说服的程度,揭示其判断受论证质量与立场偏见影响的机制。

Assessing and Explaining the Persuadability of Large Language Models as Legal Decision Tools

  • 设计三边对抗实验,量化大模型对正反方论点的响应倾向。
  • 发现模型判决易受论证质量影响,且存在明显立场偏向性。
  • 适合法律AI评估、司法自动化系统设计者参考。

随着大型语言模型(LLMs)被提议作为司法与行政场景中的法律决策辅助工具,甚至一审决策者,探究其如何回应法律问题,特别是决定疑难案件的关键因素变得至关重要。法律判决需回应各方当事人提出的主张,决策者应能有效回应,甚至可能被合理论证所说服,但不应过度受律师技巧影响而偏离案件实质。本文研究前沿开源与闭源权重大模型在法律论证中的表现,提出一种衡量三边对抗情境下模型可说服性的指标。通过原创实验,测量论证质量对模型支持某一法律观点的影响程度。进一步分析模型区分强弱论证的能力,以及位置偏差对其判断的影响。通过并行双边试验,揭示三边结构对法官模型要求的提升及其表现出的可说服性变化。最后考察影响说服力的具体因素,包括法律内容与修辞形式的相对贡献,模型判断是否匹配人类专家对论证质量的评价,以及论证数量、多样性与类型对结果的影响。研究结果对大模型在法律与行政领域的可行性具有重要启示。

原文摘要 · Abstract (English)

As Large Language Models (LLMs) are proposed as legal decision assistants, and even first-instance decision-makers, across a range of judicial and administrative contexts, it becomes essential to explore how they answer legal questions, and in particular the factors that lead them to decide difficult questions. A specific feature of legal decisions is the need to respond to arguments advanced by contending parties. A legal decision-maker must be able to engage with, and respond to, including through being potentially persuaded by, these arguments. Conversely, they should not be unduly persuadable, deciding cases based on the skills of the advocates rather than the merits of the case. In this paper we explore how frontier open- and closed-weights LLMs respond to legal arguments. We propose a metric to measure persuadability in the trilateral setting in which competing advocates seek to persuade a judge of opposite conclusions. We report original experimental results measuring how far the quality of the advocate making arguments affects the likelihood that a given model will agree with a particular legal point of view. We further examine how far models are capable of distinguishing between stronger and weaker arguments and how far model judgments in this domain are affected by positional bias. Through parallel bilateral trials we show how the trilateral setting changes the demands on judge models, and in turn their apparent persuadability. Finally we examine the specific features of arguments that affect persuasion, including the relative contribution of legal content and rhetorical form, the extent to which model persuasion tracks human expert judgments of argument quality, and the extent to which argument quantity, diversity and type affect persuasive outcomes. Our results have implications for the feasibility of adopting LLMs across legal and administrative settings.

大模型法律AI可说服性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。