用匿名学术数据估算全球高校排名,避免模型记忆干扰。
UniRank: A Multi-Agent Calibration Pipeline for Estimating University Rankings from Anonymized Bibliometric Signals
- 三阶段多智能体系统,基于匿名机构数据推断排名。
- 在THE榜单上达251.5的平均误差,零记忆指标验证真实性。
- 适合关注可解释性排名与反记忆机制的研究者。
我们提出UniRank,一个基于OpenAlex和Semantic Scholar公开文献计量数据的多智能体大模型管道,仅通过匿名机构数据估算高校在全球排名体系中的位置。系统采用三阶段架构:(a) 从匿名机构指标进行零样本估计,(b) 以真实排名高校为参照进行各系统工具增强校准,(c) 最终合成结果。关键在于所有机构信息(名称、国家、DOI、论文标题、合作国)均被脱敏,且校准过程中隐藏真实排名,防止大模型记忆干扰。在《泰晤士高等教育》世界大学排名(n=352)上,系统实现平均绝对误差251.5位,中位绝对误差131.5位,标准化平均误差12.03%,斯皮尔曼相关系数ρ=0.769,肯德尔相关系数τ=0.591,前50名命中率20.7%,前100名命中率39.8%,记忆指数为零(352所高校无任何精确匹配预测)。系统呈现系统性正偏差(+190.1位),且顶尖高校(MAE=60.5,命中率@100=90.5%)到尾部高校(MAE=328.2,命中率@100=20.8%)性能递减,表明其具备真实分析推理能力而非记忆。演示链接:https://unirank.scinito.ai
原文摘要 · Abstract (English)
We present UniRank, a multi-agent LLM pipeline that estimates university positions across global ranking systems using only publicly available bibliometric data from OpenAlex and Semantic Scholar. The system employs a three-stage architecture: (a) zero-shot estimation from anonymized institutional metrics, (b) per-system tool-augmented calibration against real ranked universities, and (c) final synthesis. Critically, institutions are anonymized -- names, countries, DOIs, paper titles, and collaboration countries are all redacted -- and their actual ranks are hidden from the calibration tools during evaluation, preventing LLM memorization from confounding results. On the Times Higher Education (THE) World University Rankings ($n=352$), the system achieves MAE = 251.5 rank positions, Median AE = 131.5, PNMAE = 12.03%, Spearman $ρ= 0.769$, Kendall $τ= 0.591$, hit rate @50 = 20.7%, hit rate @100 = 39.8%, and a Memorization Index of exactly zero (no exact-match zero-width predictions among all 352 universities). The systematic positive-signed error (+190.1 positions, indicating the system consistently predicts worse ranks than actual) and monotonic performance degradation from elite tier (MAE = 60.5, hit@100 = 90.5%) to tail tier (MAE = 328.2, hit@100 = 20.8%) provide strong evidence that the pipeline performs genuine analytical reasoning rather than recalling memorized rankings. A live demo is available at https://unirank.scinito.ai .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。