评测中文法律报告引用可信度,发现检索后还需精准筛选与应用。
LegalCiteTrust: Benchmarking Citation Trustworthiness in Chinese Long-Form Legal Research Reports
- 构建三维度可信度评估框架:存在性、忠实性、适用性。
- 现有模型引用密度高但可信度低,检索提升证据量却难提可信分。
- 需在检索后加入引用校验机制,尤其关注内容描述与适用边界。
长篇法律研究报告日益依赖大语言模型与智能研究系统,其可靠性不仅取决于任务完成度,更在于引用法律依据的可信程度。即使引用真实文献,也可能因忽略限制条件、误述权威内容或过度引申而存在风险。本文提出 LegalCiteTrust,一个用于评估中文长篇法律报告引用可信度的基准,包含72个详尽标注的报告级任务,从覆盖度、支持力和引用可信度三个维度进行评价。引用可信度通过引用层面的存在性(Existence)、忠实性(Fidelity)和适用性(Applicability)(E/F/A)量化。实验对比通用大模型、深度研究系统及法律专用系统,发现任务完成率、证据丰富度、引用密度与引用可靠性反映不同系统行为。检索工具可增强证据支持,但难以稳定提升可信度得分;基于E/F/A的修订显著改善可信度与最终评分,优于仅检查存在的过滤方式。结果表明,可信法律研究生成需在检索后实施引用感知的证据治理:系统不仅要获取法律依据,还需可靠地选择、描述与应用。
原文摘要 · Abstract (English)
Long-form legal research reports increasingly rely on LLMs and agentic research systems, but their reliability depends not only on answering the task, but also on whether cited legal authorities are trustworthy. A citation can be risky even when it points to a real source: the report may omit limiting conditions, misdescribe the authority, or use it to support a stronger claim than the source allows. We introduce LegalCiteTrust, a benchmark for evaluating citation trustworthiness in Chinese long-form legal research reports. It contains 72 densely annotated report-level tasks and evaluates reports along three dimensions: Coverage, Support, and Citation Trustworthiness. Citation Trustworthiness is operationalized through citation-level Existence, Fidelity, and Applicability (E/F/A). Experiments on general-purpose LLMs, deep-research systems, and legal-specific systems show that task completion, evidence richness, citation density, and citation reliability expose different system behaviors. Retrieval tools can improve evidence support without reliably improving the Trust score, while E/F/A-based revision improves Trust and Final score more clearly than existence-only filtering. These results suggest that trustworthy legal research generation requires citation-aware evidence governance after retrieval: systems must not only retrieve legal authorities, but also select, describe, and apply them reliably.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。