用名字得分衡量大模型对人和工具的真正记忆能力。
The Model Knows Your Project, Not You: Measuring Recognition in LLMs with NameRank
- 设计NameRank评分系统,通过真实事实判断模型是否真正记得某实体。
- 名人光环不如具体成果,诺贝尔奖得主比普通学者更易被记住。
- 模型更认知名字化的成果,而非头衔或作者名单,适合研究认知偏差者看
前沿模型在未进行检索的情况下,其权重中存储的关于人或工具的信息,常决定人类首次看到的描述。引用仅解释约三分之一的识别率,本文聚焦剩余部分,构建了[0,1]范围的NameRank识别分数:对54个群体中的4,685个实体,在36个模型上各提出一个开放式问题,由独立评委判断模型是否陈述了一个具体且不可猜测的事实(排除幻觉、上下文复述与猜测)。合成空实体得分接近零,评判结果反映实体本身而非模型。核心发现是:识别奖励的是可命名、可索引的成果,而非头衔或职称。所有奥运式荣誉均低于活跃研究者基准,因奖牌不携带名称;但在顶尖层级(诺奖、图灵、菲尔兹奖),获奖者普遍高分。对独立创作者而言,工具得分高于其创造者,传播的是命名方法或获奖论文。作为多个贡献者之一,即便在知名项目中,也几乎无法获得识别——旗舰模型报告或系统卡片上的作者大多位于得分底部。无任何文献计量指标能准确预测识别度;高密度机构在同等引用下表现更优;258个新闻事件中,识别度依赖峰值关注度而非持续性。自评实验表明,模型的自我反思基于参数化语料库,而非自身知识。
原文摘要 · Abstract (English)
What a frontier model recalls about a person or tool from its own weights -- before any retrieval step -- often shapes the first description a human sees, making that parametric corpus presence a measurement problem. Citations explain about a third of whether a model recognizes a researcher; we target the residual and build NameRank, a [0,1] recognition score: each of 4,685 entities in 54 cohorts is probed with one open-ended question across 36 models, and an independent judge returns a binary verdict against a curated gold -- did the model state a specific, non-guessable fact about this exact entity? -- so hallucination, context echo, and guesses earn nothing. Synthetic-null entities hold the floor near zero, and verdicts track the entity, not the model. One thesis organizes the findings: recognition is paid to named, indexable artifacts, not to credentials or titles. Every Olympic-style credential sits below a working-researcher baseline, because no named artifact ships with the medal, yet the ranking inverts at the marquee tier, where Nobel, Turing, and Fields laureates saturate the panel. For independent creators the tool out-ranks its maker, and the credential that does propagate is a named method or awarded paper. Being one of many named contributors to a celebrated artifact, by contrast, earns almost nothing -- the authors listed on a flagship model report or system card sit near the recognition floor -- because recognition attaches to the artifact's own distinctive name, not to the roster behind it. No bibliometric predicts recognition well; top-density institutions out-recognize peers at matched citations; and on 258 news events recognition loads on peak salience, not persistence. A self-report probe shows introspection reads a corpus prior, not its own knowledge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。