arXiv:2601.05253cs.IRcs.AI2026-01KDD

首个融合个人投票与他人预测的排名数据集,助力更精准的偏好聚合。

SP-Rank: A Dataset for Ranked Preferences with Secondary Information

  • 构建包含12000+条数据的SP-Rank,融合第一层投票与第二层预测信号。
  • 结合双信号可显著提升排名恢复准确率,优于仅用投票的方法。
  • 适合研究人类偏好建模、群体决策与人机对齐的学者使用。

我们提出SP-Rank,首个大规模公开可用的数据集,用于基准测试同时利用一阶偏好与二阶预测的排序算法。每个数据点包含个人投票(一阶信号)和对他人投票的元预测(二阶信号),支持比传统仅记录个体偏好的数据集更丰富的建模。SP-Rank覆盖地理、电影、绘画三个领域,包含超过12,000条人工生成数据,涵盖九种不同的采集格式与不同子集规模。该结构支持在专家身份未知但存在的情况下,对偏好聚合进行实证分析,其中个体投票被视为共享真实排序的噪声估计。我们通过对比仅使用一阶投票的传统聚合方法与结合双信号的SP-Voting方法,评估了其在三个核心任务上的表现:(1)完整真实排序恢复,(2)子集层级排序恢复,(3)投票者行为的概率建模。结果表明,引入二阶信号可显著提升准确性。除了社会选择外,SP-Rank还支持学习排序、从噪声众包中提取专家知识,以及基于偏好的微调中奖励模型的训练。我们已开源数据集、代码与基线评估(见https://github.com/amrit19/SP-Rank-Dataset),以推动人类偏好建模、聚合理论与人机对齐的研究。

原文摘要 · Abstract (English)

We introduce $\mathbf{SP-Rank}$, the first large-scale, publicly available dataset for benchmarking algorithms that leverage both first-order preferences and second-order predictions in ranking tasks. Each datapoint includes a personal vote (first-order signal) and a meta-prediction of how others will vote (second-order signal), allowing richer modeling than traditional datasets that capture only individual preferences. SP-Rank contains over 12,000 human-generated datapoints across three domains -- geography, movies, and paintings, and spans nine elicitation formats with varying subset sizes. This structure enables empirical analysis of preference aggregation when expert identities are unknown but presumed to exist, and individual votes represent noisy estimates of a shared ground-truth ranking. We benchmark SP-Rank by comparing traditional aggregation methods that use only first-order votes against SP-Voting, a second-order method that jointly reasons over both signals to infer ground-truth rankings. While SP-Rank also supports models that rely solely on second-order predictions, our benchmarks emphasize the gains from combining both signals. We evaluate performance across three core tasks: (1) full ground-truth rank recovery, (2) subset-level rank recovery, and (3) probabilistic modeling of voter behavior. Results show that incorporating second-order signals substantially improves accuracy over vote-only methods. Beyond social choice, SP-Rank supports downstream applications in learning-to-rank, extracting expert knowledge from noisy crowds, and training reward models in preference-based fine-tuning pipelines. We release the dataset, code, and baseline evaluations (available at https://github.com/amrit19/SP-Rank-Dataset ) to foster research in human preference modeling, aggregation theory, and human-AI alignment.

偏好建模数据集排序人机对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。