轻量高效强化学习超参优化框架，单次运行即完成调优。
ULTHO: Ultra-Lightweight yet Efficient Hyperparameter Optimization in Deep Reinforcement Learning
- 将超参优化建模为带聚类的多臂老虎机问题，直接关联长期回报。
- 在ALE、Procgen等基准上实现优于现有方法的性能，仅需单次训练。
- 适合追求快速部署与自动化调优的强化学习研究者与工程师。
超参数优化（HPO）是机器学习中的关键挑战，显著影响训练效率与模型性能。然而，在深度强化学习（RL）中实现高效且稳健的HPO始终困难，主要因高非平稳性与计算开销。现有方法尝试将种群基于训练或贝叶斯优化等通用技术适配到RL场景，但仍存在样本效率低、计算成本高等问题，难以广泛落地。本文提出ULTHO，一种超轻量但高效的单次运行深度强化学习超参优化框架。具体地，将HPO过程建模为带有聚类臂的多臂老虎机（MABC），并将其与长期回报优化直接关联。ULTHO还提供量化与统计视角，实现高效超参筛选。我们在ALE、Procgen、MiniGrid和PyBullet等多个基准上进行测试。大量实验表明，ULTHO在简单架构下即可取得优异性能，推动先进自动化强化学习系统的发展。
原文摘要 · Abstract (English)
Hyperparameter optimization (HPO) is a billion-dollar problem in machine learning, which significantly impacts the training efficiency and model performance. However, achieving efficient and robust HPO in deep reinforcement learning (RL) is consistently challenging due to its high non-stationarity and computational cost. To tackle this problem, existing approaches attempt to adapt common HPO techniques (e.g., population-based training or Bayesian optimization) to the RL scenario. However, they remain sample-inefficient and computationally expensive, which cannot facilitate a wide range of applications. In this paper, we propose ULTHO, an ultra-lightweight yet powerful framework for fast HPO in deep RL within single runs. Specifically, we formulate the HPO process as a multi-armed bandit with clustered arms (MABC) and link it directly to long-term return optimization. ULTHO also provides a quantified and statistical perspective to filter the HPs efficiently. We test ULTHO on benchmarks including ALE, Procgen, MiniGrid, and PyBullet. Extensive experiments demonstrate that the ULTHO can achieve superior performance with a simple architecture, contributing to the development of advanced and automated RL systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。