arXiv:2505.02010cs.NEcs.LG2025-05ICML被引 13

用离线学习提升黑箱优化的配置效率,兼顾性能与速度。

Meta-Black-Box-Optimization through Offline Q-function Learning

  • 将算法配置转化为长序列决策,设计分解式Q函数降低学习难度。
  • 在离线数据上实现稳定训练,性能优于现有在线/离线方法。
  • 结合Mamba架构加速长序列学习,适合大规模优化任务场景。

近期元黑箱优化(MetaBBO)进展表明,利用强化学习学习元策略以动态配置优化算法,在任务分布上可显著提升底层黑箱优化算法性能。然而现有方法依赖在线学习,导致效率低下。为此,本文提出基于离线学习的MetaBBO框架Q-Mamba,兼顾有效性与效率。首先将动态算法配置(DAC)任务建模为长序列决策过程,并引入有效的Q函数分解机制,降低复杂配置空间中的学习难度。在此基础上,提出三项新设计:一是构建兼具探索与利用平衡的离线DAC经验数据集收集策略;二是建立基于分解的保守Q损失函数,促进从离线数据中稳定学习;三是引入Mamba架构,通过选择性状态建模和硬件感知并行扫描,提升长序列学习的效率与效果。大量实验表明,Q-Mamba在性能上达到甚至超越现有在线/离线基线,同时显著提升训练效率。代码已开源:https://github.com/MetaEvo/Q-Mamba。

原文摘要 · Abstract (English)

Recent progress in Meta-Black-Box-Optimization (MetaBBO) has demonstrated that using RL to learn a meta-level policy for dynamic algorithm configuration (DAC) over an optimization task distribution could significantly enhance the performance of the low-level BBO algorithm. However, the online learning paradigms in existing works makes the efficiency of MetaBBO problematic. To address this, we propose an offline learning-based MetaBBO framework in this paper, termed Q-Mamba, to attain both effectiveness and efficiency in MetaBBO. Specifically, we first transform DAC task into long-sequence decision process. This allows us further introduce an effective Q-function decomposition mechanism to reduce the learning difficulty within the intricate algorithm configuration space. Under this setting, we propose three novel designs to meta-learn DAC policy from offline data: we first propose a novel collection strategy for constructing offline DAC experiences dataset with balanced exploration and exploitation. We then establish a decomposition-based Q-loss that incorporates conservative Q-learning to promote stable offline learning from the offline dataset. To further improve the offline learning efficiency, we equip our work with a Mamba architecture which helps long-sequence learning effectiveness and efficiency by selective state model and hardware-aware parallel scan respectively. Through extensive benchmarking, we observe that Q-Mamba achieves competitive or even superior performance to prior online/offline baselines, while significantly improving the training efficiency of existing online baselines. We provide sourcecodes of Q-Mamba at https://github.com/MetaEvo/Q-Mamba.

元学习黑箱优化离线RLMamba

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。