用嵌入和图神经网络识别大模型生成的参考文献，准确率达93%。
Structurally Human, Semantically Biased: Detecting LLM-Generated References with Embeddings and GNNs
- 基于标题摘要嵌入和图结构特征，构建检测模型。
- 嵌入特征使检测准确率提升至93%，远超仅用结构特征。
- 适合关注AI生成内容检测与学术诚信的研究者。
大型语言模型被广泛用于整理参考文献，引发疑问：其生成的参考文献列表是否可与人类撰写者区分？研究构建了10,000篇核心论文（约27.5万条引用）的配对引文图，包含真实引用与GPT-4o生成的引用，并引入领域匹配的随机基线（保留出度和领域分布但破坏潜在结构）。比较仅使用结构特征（度、接近度、特征向量中心性、聚类系数、边数）与3072维标题/摘要嵌入的效果，采用随机森林（RF）分析图级聚合特征，以及结合节点特征的图神经网络（GNN）。仅用结构特征时，区分效果较差（RF准确率≈0.60），虽能有效排除随机基线（≈0.89–0.92）。而嵌入特征显著提升区分能力：基于嵌入的RF达到≈0.83准确率，使用嵌入作为节点特征的GNN在测试集上实现93%准确率。通过使用Claude Sonnet 4.5及多个嵌入模型（OpenAI、SPECTER）复现，验证结果稳健：真实引用与Claude生成引用的区分度为≈0.77，且能干净地拒绝随机基线。结论表明，仅凭参数知识生成的参考文献在引用拓扑上高度模仿人类行为，但留下可检测的语义痕迹；检测与去偏应聚焦内容信号而非全局图结构。
原文摘要 · Abstract (English)
Large language models are increasingly used to curate bibliographies, raising the question: are their reference lists distinguishable from human ones? We build paired citation graphs, ground truth and GPT-4o-generated (from parametric knowledge), for 10,000 focal papers ($\approx$ 275k references) from SciSciNet, and added a field-matched random baseline that preserves out-degree and field distributions while breaking latent structure. We compare (i) structure-only node features (degree/closeness/eigenvector centrality, clustering, edge count) with (ii) 3072-D title/abstract embeddings, using an RF on graph-level aggregates and Graph Neural Networks with node features. Structure alone barely separates GPT from ground truth (RF accuracy $\approx$ 0.60) despite cleanly rejecting the random baseline ($\approx$ 0.89--0.92). By contrast, embeddings sharply increase separability: RF on aggregated embeddings reaches $\approx$ 0.83, and GNNs with embedding node features achieve 93\% test accuracy on GPT vs.\ ground truth. We show the robustness of our findings by replicating the pipeline with Claude Sonnet 4.5 and with multiple embedding models (OpenAI and SPECTER), with RF separability for ground truth vs.\ Claude $\approx 0.77$ and clean rejection of the random baseline. Thus, LLM bibliographies, generated purely from parametric knowledge, closely mimic human citation topology, but leave detectable semantic fingerprints; detection and debiasing should target content signals rather than global graph structure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。