arXiv:2605.12382cs.CL2026-05中稿 · SIGIR 2026被引 1

大模型偏爱热门实体,主要因为训练时接触得多。

Pretraining Exposure Explains Popularity Judgments in Large Language Models

论文配图:Pretraining Exposure Explains Popularity Judgments in Large Language Models
图 1 · 摘自论文原文
  • 用公开数据集分析模型训练中对实体的接触频率
  • 模型偏好与训练暴露度相关性高于真实网络热度
  • 大模型和长尾实体更依赖训练数据暴露程度

大型语言模型(LLMs)对知名实体存在系统性偏好,通常归因于流行度偏差。然而,这种偏好是源于现实世界中的真实流行度,还是预训练过程中的统计暴露,尚不明确,主要受限于多数训练语料库的不可访问性。本文首次基于完全可观察的预训练数据进行大规模直接分析。利用开放的OLMo模型及其完整预训练语料库Dolma,我们在7.4万亿个词元上计算了实体级别的精确暴露统计。分析涵盖2000个实体,涉及人物、地点、组织、艺术作品、产品五类,并将预训练暴露度与维基百科页面浏览量及两种由模型生成的流行度信号(直接标量估计与成对比较)进行对比。结果表明,预训练暴露度与维基百科流行度高度相关,验证了暴露度作为训练期真实显著性的有效代理。更重要的是,我们发现模型的流行度判断与暴露度的相关性高于与维基百科热度的相关性,尤其是在通过成对比较获取信号时。该关联在大模型中最强,并在长尾区域持续存在,而维基百科热度在此处变得不可靠。总体而言,研究证实大模型的流行度先验主要由预训练统计数据塑造,而非外部流行信号,为数据暴露在驱动流行度偏差中的核心作用提供了实证支持。

原文摘要 · Abstract (English)

Large language models (LLMs) exhibit systematic preferences for well-known entities, a phenomenon often attributed to popularity bias. However, the extent to which these preferences reflect real-world popularity versus statistical exposure during pretraining remains unclear, largely due to the inaccessibility of most training corpora. We provide the first direct, large-scale analysis of popularity bias grounded in fully observable pretraining data. Leveraging the open OLMo models and their complete pretraining corpus, Dolma, we compute precise entity-level exposure statistics across 7.4 trillion tokens. We analyze 2,000 entities spanning five types (Person, Location, Organization, Art, Product) and compare pretraining exposure against Wikipedia pageviews and two elicited LLM popularity signals: direct scalar estimation and pairwise comparison. Our results show that pretraining exposure strongly correlates with Wikipedia popularity, validating exposure as a meaningful proxy for real-world salience during the training period. More importantly, we find that LLM popularity judgments align more closely with exposure than with Wikipedia, especially when elicited via pairwise comparisons. This alignment is strongest for larger models and persists in the long tail, where Wikipedia popularity becomes unreliable. Overall, our findings demonstrate that popularity priors in LLMs are primarily shaped by pretraining statistics rather than external popularity signals, offering concrete evidence that data exposure plays a central role in driving popularity bias.

大模型流行度偏差预训练数据暴露度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。