用大模型自动发现优化算法的奖励函数，提升性能并减少人工设计偏差。
READY: Reward Discovery for Meta-Black-Box Optimization
- 用大模型结合演化策略自动搜索最优奖励函数
- 发现的奖励函数可显著提升现有元黑箱优化方法性能
- 支持多任务并行进化，加速奖励发现且共享知识
元黑箱优化（MetaBBO）是优化领域新兴方向，可通过强化学习元学习算法设计策略以提升优化性能。目前，已有研究中的奖励函数均由人工设计，存在设计偏见和奖励欺骗风险。本文提出利用大语言模型（LLM）作为自动化奖励发现工具。在有效性方面，借鉴启发式演化的思想，在迭代式LLM程序搜索中引入定制化演化范式，确保持续改进；在效率方面，引入多任务演化架构，支持对多种元黑箱优化方法并行进行奖励发现，并通过任务间知识共享加速收敛。实验表明，本方法发现的奖励函数能有效提升现有元黑箱优化方法的表现，凸显了奖励设计在元黑箱优化中的关键作用。项目代码已公开于https://anonymous.4open.science/r/ICML_READY-747F。
原文摘要 · Abstract (English)
Meta-Black-Box Optimization (MetaBBO) is an emerging avenue within Optimization community, where algorithm design policy could be meta-learned by reinforcement learning to enhance optimization performance. So far, the reward functions in existing MetaBBO works are designed by human experts, introducing certain design bias and risks of reward hacking. In this paper, we use Large Language Model~(LLM) as an automated reward discovery tool for MetaBBO. Specifically, we consider both effectiveness and efficiency sides. On effectiveness side, we borrow the idea of evolution of heuristics, introducing tailored evolution paradigm in the iterative LLM-based program search process, which ensures continuous improvement. On efficiency side, we additionally introduce multi-task evolution architecture to support parallel reward discovery for diverse MetaBBO approaches. Such parallel process also benefits from knowledge sharing across tasks to accelerate convergence. Empirical results demonstrate that the reward functions discovered by our approach could be helpful for boosting existing MetaBBO works, underscoring the importance of reward design in MetaBBO. We provide READY's project at https://anonymous.4open.science/r/ICML_READY-747F.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。