arXiv:2506.05634cs.LGcs.AI2025-06中稿 · ICLR被引 4

自动发现多样高质策略,无需预设行为描述符。

AutoQD: Automatic Discovery of Diverse Behaviors with Quality-Diversity Optimization

  • 用占用测度嵌入生成行为描述符,避免人工设计。
  • 在连续控制任务中成功发现多样策略,性能优于基线。
  • 适合无监督强化学习与开放世界行为探索场景。

质量-多样性(QD)算法在发现多样且高性能解方面表现优异,但依赖于手工设计的行为描述符,限制了探索范围。我们基于策略与占用测度的等价性,提出一种理论支撑的方法:通过将马尔可夫决策过程中的策略占用测度进行嵌入,自动生成行为描述符。AutoQD利用随机傅里叶特征近似策略占用测度间的最大均值差异(MMD),其嵌入距离反映真实行为差异。对这些嵌入进行低维投影,提取最具行为意义的维度,作为CMA-MAE——一种先进的黑箱QD方法——的行为描述符,以发现多样化策略。我们证明,随着采样轨迹数和嵌入维度增加,嵌入距离收敛至真实的MMD距离。在多个连续控制任务中,AutoQD无需预定义行为描述符即可发现多样化策略,为无监督强化学习与QD优化提供了有力替代方案。该方法为序列决策中的开放式学习与自动化行为发现开辟新路径,无需领域知识。代码已开源:https://github.com/conflictednerd/autoqd-code。

原文摘要 · Abstract (English)

Quality-Diversity (QD) algorithms have shown remarkable success in discovering diverse, high-performing solutions, but rely heavily on hand-crafted behavioral descriptors that constrain exploration to predefined notions of diversity. Leveraging the equivalence between policies and occupancy measures, we present a theoretically grounded approach to automatically generate behavioral descriptors by embedding the occupancy measures of policies in Markov Decision Processes. Our method, AutoQD, leverages random Fourier features to approximate the Maximum Mean Discrepancy (MMD) between policy occupancy measures, creating embeddings whose distances reflect meaningful behavioral differences. A low-dimensional projection of these embeddings that captures the most behaviorally significant dimensions can then be used as behavioral descriptors for CMA-MAE, a state of the art blackbox QD method, to discover diverse policies. We prove that our embeddings converge to true MMD distances between occupancy measures as the number of sampled trajectories and embedding dimensions increase. Through experiments in multiple continuous control tasks we demonstrate AutoQD's ability in discovering diverse policies without predefined behavioral descriptors, presenting a well-motivated alternative to prior methods in unsupervised Reinforcement Learning and QD optimization. Our approach opens new possibilities for open-ended learning and automated behavior discovery in sequential decision making settings without requiring domain-specific knowledge. Source code is available at https://github.com/conflictednerd/autoqd-code.

强化学习质量多样性自动发现策略探索

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。