提升强化学习超参优化效率,让调参更透明可解释
Hyperparameter Optimisation with Practical Interpretability and Explanation Methods in Probabilistic Curriculum Learning
- 用树形结构搜索算法精简超参空间,加速优化过程
- 在点迷宫和电机控制任务中验证性能提升效果
- 结合SHAP方法揭示超参影响机制,适合工程落地
超参数优化(HPO)对强化学习(RL)性能至关重要,因RL算法对超参设置极为敏感。概率课程学习(PCL)通过结构化学习过程提升RL表现,但有效调参仍具挑战且计算成本高。本文基于AlgOS框架与Optuna的树状结构帕岑估计器(TPE),对标准RL任务(包括点迷宫导航与直流电机控制)中的超参交互及其对PCL算法性能的影响进行实证分析。提出优化超参搜索空间的策略,显著提升优化效率。同时引入一种针对超参影响的新型SHAP可解释方法,清晰揭示各超参及其交互作用对性能的影响。本工作提供实用指南与可解释工具,大幅增强强化学习中超参优化的有效性与计算可行性。
原文摘要 · Abstract (English)
Hyperparameter optimisation (HPO) is crucial for achieving strong performance in reinforcement learning (RL), as RL algorithms are inherently sensitive to hyperparameter settings. Probabilistic Curriculum Learning (PCL) is a curriculum learning strategy designed to improve RL performance by structuring the agent's learning process, yet effective hyperparameter tuning remains challenging and computationally demanding. In this paper, we provide an empirical analysis of hyperparameter interactions and their effects on the performance of a PCL algorithm within standard RL tasks, including point-maze navigation and DC motor control. Using the AlgOS framework integrated with Optuna's Tree-Structured Parzen Estimator (TPE), we present strategies to refine hyperparameter search spaces, enhancing optimisation efficiency. Additionally, we introduce a novel SHAP-based interpretability approach tailored specifically for analysing hyperparameter impacts, offering clear insights into how individual hyperparameters and their interactions influence RL performance. Our work contributes practical guidelines and interpretability tools that significantly improve the effectiveness and computational feasibility of hyperparameter optimisation in reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。