arXiv:2510.25378cs.CLcs.AI2025-10被引 3

高被引论文更少被幻觉,因训练数据重复度高,模型直接记忆。

Hallucinations in Bibliographic Recommendation: Citation Frequency as a Proxy for Training Data Redundancy

  • 用引用频次衡量训练数据重复度,研究其对幻觉的影响。
  • 引用超1000次的论文几乎完全记忆,幻觉率极低。
  • 适合关注LLM知识来源与可信推荐的研究者。

大语言模型在文献推荐中存在生成不存在论文的幻觉问题。本研究假设模型正确生成文献信息的能力取决于知识是生成还是记忆,而高被引论文(即在训练语料中频繁出现)的幻觉率更低。因此,将引用次数视为训练数据冗余的代理指标,探究其对模型输出幻觉文献的影响。使用GPT-4.1在二十个计算机科学领域生成并人工验证了100条文献记录,通过生成与真实元数据的余弦相似度评估事实一致性。结果表明:(i) 幻觉率在不同研究领域间存在差异;(ii) 引用次数与事实准确性显著相关;(iii) 当引用次数超过约1000次时,文献信息近乎原样记忆。这说明高被引论文几乎被模型完整保留,揭示了从泛化到记忆的临界点。

原文摘要 · Abstract (English)

Large language models (LLMs) have been increasingly applied to a wide range of tasks, from natural language understanding to code generation. While they have also been used to assist in bibliographic recommendation, the hallucination of non-existent papers remains a major issue. Building on prior studies, this study hypothesizes that an LLM's ability to correctly produce bibliographic information depends on whether the underlying knowledge is generated or memorized, with highly cited papers (i.e., more frequently appear in the training corpus) showing lower hallucination rates. We therefore assume citation count as a proxy for training data redundancy (i.e., the frequency with which a given bibliographic record is repeatedly represented in the pretraining corpus) and investigate how citation frequency affects hallucinated references in LLM outputs. Using GPT-4.1, we generated and manually verified 100 bibliographic records across twenty computer-science domains, and measured factual consistency via cosine similarity between generated and authentic metadata. The results revealed that (i) hallucination rates vary across research domains, (ii) citation count is strongly correlated with factual accuracy, and (iii) bibliographic information becomes almost verbatimly memorized beyond approximately 1,000 citations. These findings suggest that highly cited papers are nearly verbatimly retained in the model, indicating a threshold where generalization shifts into memorization.

文献推荐幻觉分析引用频率模型记忆

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。