用机器学习判断随机优化实验要跑多少次才可靠。
Learning to Assess the Reliability of Number-of-Runs Estimation in Stochastic Optimization
- 基于23个无能耗特征训练分类器，判断实验次数估计是否可靠
- 在13.2万次实验数据上实现高召回率检测错误估计
- 适合做算法对比实验的可靠性验证，尤其关注少数错误案例
在大规模随机优化算法基准测试中，关键问题已从‘是否需要重复运行’转变为‘如何确定足够证据而不浪费计算成本’。本文研究了一种基于学习的改进方法，扩展了近期一种基于异常值处理和偏度对称性检查的在线启发式算法。利用132,000次Nevergrad在COCO上的运行结果（24个问题，每维20维，每个问题10个实例，共11种优化器），我们基于23个统计、能量无关、形状与稳定性特征训练分类器，预测运行次数估计的可靠性，重点提升对错误估计的少数类召回率。在同配置学习设置下评估模型性能，即模型在相同优化器的数据上训练与测试。结果显示，运行次数可靠性可在同配置场景下被有效学习，能以高少数类召回率识别不可靠估计，但受限于固定配置内的数据多样性，性能仍有局限。
原文摘要 · Abstract (English)
In large-scale benchmarking of stochastic optimization algorithms, the key challenge is no longer whether repeated runs are needed for reliability, but how to determine when sufficient evidence has been collected without incurring unnecessary computational cost. We study a learning-based extension of a recent empirical online heuristic that adaptively estimates the required number of runs using outlier handling and skewness-based symmetry checks. Using annotated outcomes from 132{,}000 Nevergrad runs on COCO (24 problems in 20 dimensions, 10 instances each, 11 optimizers), we train classifiers on 23 statistical, energy-free, and shape and stability features to predict whether a run-number estimate is reliable, prioritizing detection of incorrect estimates via minority-class recall. We evaluate reliability prediction using a within-configuration learning setup, where models are trained and tested on data sharing the same optimizer. The results show that run-number reliability can be learned in a within-configuration scenario, enabling detection of unreliable estimates with high minority-class recall, although performance remains limited by the restricted data diversity within fixed configurations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。