发现文本嵌入的相似性无法准确匹配研究议题,影响科学检索效果。
Topic Is Not Agenda: A Citation-Community Audit of Text Embeddings
- 构建358万论文的引用图,按领域和研究议题分层评估嵌入效果。
- 90%检索结果与查询议题不符,四大模型在议题层级表现均差。
- 引用计数重排优于嵌入相似性,揭示嵌入遗漏了关键议题信号。
向量搜索与检索增强生成(RAG)依赖文本嵌入间的余弦相似性反映概念相关性。我们检验该假设的失效边界。构建覆盖358万篇科学论文的增强引用图,并使用Leiden CPM算法在两个粒度下划分:子领域(L1)与研究议题(L2,L1内分层)。四种先进嵌入模型(Gemini、Qwen3-8B、Qwen3-0.6B、SPECTER2)在L1层级表现尚可(前10名同领域率45%-52%),但在L2层级显著失效:前10名邻居仅15%-21%共享查询的研究议题。绝对意义上,每10篇检索结果中有8篇偏离议题。该失败现象横跨八个科学领域且普遍存在于所有四类模型中;尽管SPECTER2采用基于引用的对比训练,表现最弱。作为诊断工具,我们测试该图结构本身能否作为检索信号:一个简单的引用计数重排策略,在80个精心设计的议题查询上,结合LLM扩展的布尔检索可达57.7%的前1名正确率,结合原始BM25可达59.6%,分别比最佳余弦检索器(Gemini,50.6%)高出约9个百分点,比纯BM25(39.3%)高出20个百分点。该探针捕捉到图结构中嵌入所遗漏的议题匹配信号,将近期关于单向量检索的理论极限与科学RAG中的具体失效模式联系起来。
原文摘要 · Abstract (English)
Vector search and retrieval-augmented generation (RAG) rest on the assumption that cosine similarity between text embeddings reflects conceptual relatedness. We measure where this assumption breaks. We build an augmented citation graph over 3.58M scientific papers and partition it via Leiden CPM at two granularities: sub-field (L1) and research-agenda (L2, hierarchical inside each L1). Four state-of-the-art embeddings (Gemini, Qwen3-8B, Qwen3-0.6B, SPECTER2) clear the L1 bar reasonably (45-52% top-10 same-rate) but stop working at L2: only 15-21% of top-10 neighbors share the query's research agenda. In absolute terms, 8 of every 10 retrieved papers are off-agenda. The failure is universal across eight scientific domains and all four models; SPECTER2, despite its citation-based contrastive training, is the weakest. As a diagnostic probe, we test whether the same augmented graph also functions as a retrieval signal: a deliberately simple citation-count rerank reaches 57.7% top-1 L2 on top of LLM-expanded Boolean retrieval and 59.6% on top of plain BM25, on 80 curated agenda queries -- about 9 points above the best cosine retriever (Gemini, 50.6%) and 20 points above BM25 alone (39.3%). The probe isolates a slice of the agenda-matching signal the graph carries but the embeddings miss, connecting recent theoretical limits on single-vector retrieval to a concrete failure mode of scientific RAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。