新算法RASDA在超算上实现高效超参优化，兼顾速度与精度。
Resource-Adaptive Successive Doubling for Hyperparameter Optimization with Large Datasets on High-Performance Computing Systems
- 动态调整训练资源分配，结合多卡并行与训练轮数自适应策略
- 在1024块GPU上运行，相比ASHA提速最高达1.9倍
- 首次用于千兆级科学数据集的系统性超参优化，适合大规模科研场景
在高性能计算（HPC）系统中，可通过并行评估多个超参数配置加速超参数优化（HPO）。现有先进方法基于博弈论思想，采用逐次减半策略，通过低精度性能指标预估最终表现，并随时间向更优配置分配更多资源。通常以训练轮数或工作进程数作为资源，让更有潜力的配置获得更长训练时间或更多并行计算单元。本文提出新型资源自适应逐次加倍算法（RASDA），融合资源自适应机制与异步逐次减半算法（ASHA）。该方法在现代HPC系统上扩展至最多1024个图形处理单元（GPU），应用于计算机视觉（CV）、计算流体力学（CFD）及增材制造（AM）领域的多种神经网络模型，在大型数据集上进行训练。由于完整训练成本过高，常规方法难以执行多次全量训练。实验表明，RASDA在运行时间上相较ASHA最高提升1.9倍，同时保持甚至超越了最终模型的性能。借助RASDA，首次在文献中实现了对千兆级科学数据集的系统性超参数优化，显著提升了复杂模型在海量科学数据上的高效调优能力。RASDA代码已开源：https://github.com/olympiquemarcel/rasda
原文摘要 · Abstract (English)
On High-Performance Computing (HPC) systems, several hyperparameter configurations can be evaluated in parallel to speed up the Hyperparameter Optimization (HPO) process. State-of-the-art HPO methods follow a bandit-based approach and build on top of successive halving, where the final performance of a combination is estimated based on a lower than fully trained fidelity performance metric and more promising combinations are assigned more resources over time. Frequently, the number of epochs is treated as a resource, letting more promising combinations train longer. Another option is to use the number of workers as a resource and directly allocate more workers to more promising configurations via data-parallel training. This article proposes a novel Resource-Adaptive Successive Doubling Algorithm (RASDA), which combines a resource-adaptive successive doubling scheme with the plain Asynchronous Successive Halving Algorithm (ASHA). Scalability of this approach is shown on up to 1,024 Graphics Processing Units (GPUs) on modern HPC systems. It is applied to different types of Neural Networks (NNs) and trained on large datasets from the Computer Vision (CV), Computational Fluid Dynamics (CFD), and Additive Manufacturing (AM) domains, where performing more than one full training run is usually infeasible. Empirical results show that RASDA outperforms ASHA by a factor of up to 1.9 with respect to the runtime. At the same time, the solution quality of final ASHA models is maintained or even surpassed by the implicit batch size scheduling of RASDA. With RASDA, systematic HPO is applied to a terabyte-scale scientific dataset for the first time in the literature, enabling efficient optimization of complex models on massive scientific data. The implementation of RASDA is available on https://github.com/olympiquemarcel/rasda
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。