发现大模型常虚构引用链接，提出工具可让错误率降为1%以下。
Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agents
- 用网页存档检测引用链接真伪，区分伪造与失效链接
- 3%~13%引用链接根本不存在，5%~18%无法打开
- 工具可大幅减少错误引用，适合研究型AI开发者使用
大型语言模型和深度研究代理会提供引用链接支持其说法，但这些链接的可靠性尚未系统评估。我们基于DRBench（53,090个链接）和ExpertQA（168,021个链接，覆盖32个学科）对10个模型和代理进行了六项研究，发现3%–13%的引用链接是幻觉——在Wayback Machine中无记录，可能从未存在；5%–18%的链接无法访问。深度研究代理每查询生成的链接远多于搜索增强型模型，但幻觉率更高。不同领域差异显著：非解析率从5.4%（商科）到11.4%（神学）不等，模型间差异更大。故障分解显示，部分模型完全虚构所有失效链接，另一些则显示大量链接腐烂，反映真实检索过程。为此，我们发布开源工具urlhealth，利用Wayback Machine实现链接存活检测与失效类型分类。在代理自纠错实验中，配备该工具的模型将不可访问链接减少6–79倍，降至1%以下，但效果取决于模型调用工具的能力。数据与工具均已公开。我们的分析结果、失败分类体系及开源工具表明，引用链接有效性既可大规模测量，也可实际修复。
原文摘要 · Abstract (English)
Large language models and deep research agents supply citation URLs to support their claims, yet the reliability of these citations has not been systematically measured. We address six research questions about citation URL validity using 10 models and agents on DRBench (53,090 URLs) and 3 models on ExpertQA (168,021 URLs across 32 academic fields). We find that 3--13\% of citation URLs are hallucinated -- they have no record in the Wayback Machine and likely never existed -- while 5--18\% are non-resolving overall. Deep research agents generate substantially more citations per query than search-augmented LLMs but hallucinate URLs at higher rates. Domain effects are pronounced: non-resolving rates range from 5.4\% (Business) to 11.4\% (Theology), with per-model effects even larger. Decomposing failures reveals that some models fabricate every non-resolving URL, while others show substantial link-rot fractions indicating genuine retrieval. As a solution, we release urlhealth, an open-source tool for URL liveness checking and stale-vs-hallucinated classification using the Wayback Machine. In agentic self-correction experiments, models equipped with urlhealth reduce non-resolving citation URLs by $6\textrm{--}79\times$ to under 1\%, though effectiveness depends on the model's tool-use competence. The tool and all data are publicly available. Our characterization findings, failure taxonomy, and open-source tooling establish that citation URL validity is both measurable at scale and correctable in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。