用完整排名信息预测算法未来胜率，比只看胜场更准。
Who is the Winning Algorithm? Rank Aggregation for Comparative Studies
- 基于所有排名名次（非仅胜场）建模算法表现
- 在合成与真实数据集上显著优于现有方法
- 适合做算法比较研究的科研人员参考
考虑一组共 m 个竞争的机器学习算法。已知它们在基准数据集上的表现，我们希望识别出最有可能在未来未见数据集上排名第一的算法。标准最大似然方法仅统计每个算法的胜场数。本文认为，完整的排名信息（如各算法获得第2、第3名的次数）蕴含更多有效信息。然而，如何有效利用这些信息仍不明确。为此，本文提出一种新颖的概念框架，基于算法在基准数据集上的完整排名，估计其获胜概率。该方法在合成数据和真实世界案例中均显著优于现有方法。
原文摘要 · Abstract (English)
Consider a collection of m competing machine learning algorithms. Given their performance on a benchmark of datasets, we would like to identify the best performing algorithm. Specifically, which algorithm is most likely to ``win'' (rank highest) on a future, unseen dataset. The standard maximum likelihood approach suggests counting the number of wins per each algorithm. In this work, we argue that there is much more information in the complete rankings. That is, the number of times that each algorithm finished second, third and so forth. Yet, it is not entirely clear how to effectively utilize this information for our purpose. In this work we introduce a novel conceptual framework for estimating the win probability for each of the m algorithms, given their complete rankings over a benchmark of datasets. Our proposed framework significantly improves upon currently known methods in synthetic and real-world examples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。