arXiv:2602.23630cs.LG2026-02KDD被引 1

通过诊断训练问题提前淘汰无效超参配置,大幅节省深度学习调优时间。

BTTackler: A Diagnosis-based Framework for Efficient Deep Learning Hyperparameter Optimization

  • 引入训练诊断机制,用量化指标自动识别梯度消失等训练异常
  • 相比基线方法平均节省40.33%时间,相同时间内多完成44.5%优质试验
  • 适用于追求高效自动化调参的研究者与工程团队

深度学习超参数优化(HPO)成本高昂,现有自动化方法多依赖准确率指标指导搜索,但早期难以捕捉梯度消失、收敛不足等训练问题,导致大量计算资源浪费于无效试验。本文提出新型框架BTTackler,通过计算一组精心设计的量化指标自动诊断训练状态,一旦发现异常即提前终止。在三种经典深度神经网络和四种主流HPO方法上验证,结果表明:(1) 平均减少40.33%时间即可达到与基线相当的准确率;(2) 在相同时间预算内,平均多完成44.5%的优质超参试验。我们还开源了Python库,支持用户以极小代码改动集成BTTackler至自动化调参流程。

原文摘要 · Abstract (English)

Hyperparameter optimization (HPO) is known to be costly in deep learning, especially when leveraging automated approaches. Most of the existing automated HPO methods are accuracy-based, i.e., accuracy metrics are used to guide the trials of different hyperparameter configurations amongst a specific search space. However, many trials may encounter severe training problems, such as vanishing gradients and insufficient convergence, which can hardly be reflected by accuracy metrics in the early stages of the training and often result in poor performance. This leads to an inefficient optimization trajectory because the bad trials occupy considerable computation resources and reduce the probability of finding excellent hyperparameter configurations within a time limitation. In this paper, we propose \textbf{Bad Trial Tackler (BTTackler)}, a novel HPO framework that introduces training diagnosis to identify training problems automatically and hence tackles bad trials. BTTackler diagnoses each trial by calculating a set of carefully designed quantified indicators and triggers early termination if any training problems are detected. Evaluations are performed on representative HPO tasks consisting of three classical deep neural networks (DNN) and four widely used HPO methods. To better quantify the effectiveness of an automated HPO method, we propose two new measurements based on accuracy and time consumption. Results show the advantage of BTTackler on two-fold: (1) it reduces 40.33\% of time consumption to achieve the same accuracy comparable to baseline methods on average and (2) it conducts 44.5\% more top-10 trials than baseline methods on average within a given time budget. We also released an open-source Python library that allows users to easily apply BTTackler to automated HPO processes with minimal code changes.

超参优化训练诊断效率提升自动化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。