用记忆引导的代码策略进化框架,提升长程任务执行成功率。
MEMENTO: Memory-Guided Memetic Code-as-Policy Evolution

- 基于记忆引导的局部优化与遗传变异,逐步改进策略代码
- 在两个真实场景中成功解决复杂操作任务,泛化能力更强
- 支持从仿真到物理机器人的直接部署,适合机器人控制研究者
长时序具身任务需要在任务完成前执行多个依赖动作。将策略表示为可执行程序(代码即策略)可使决策逻辑在回放评估后被检查和修改。修改后的程序可通过回放性能重新执行并比较,将策略改进视为执行引导的程序搜索。由大语言模型驱动的进化方法能自然实现这一搜索过程,通过生成变体并选择高性能候选者。然而,现有方法主要依赖独立生成的变体,缺乏序列化的局部优化阶段。我们提出 MEMENTO,一种记忆引导的单精英遗传框架用于代码即策略的进化。MEMENTO 首先进化一个回放评估器,将策略回放映射为标量适应度和结构化反馈指标。适应度用于选择接受的候选者及下一精英,反馈指标则指导由记忆引导的爬山、宏突变和交叉生成的策略提案。我们在两个长时序具身任务域上评估:Robosuite Franka Tower-of-Hanoi 操控任务与 AI2-THOR 家居交互任务。MEMENTO 在任务成功率和对未见的 Robosuite 物体配置及 AI2-THOR 场景的泛化能力上均优于 Eureka 与 REvolve(经适配为代码即策略的进化基线)。消融实验表明,零样本生成与未进化的评估器无法解决任一领域,且移除策略搜索分支会降低性能。最后,我们将最优进化的 Robosuite 策略部署于真实 Franka 机器人上,验证了该代码即策略从仿真到现实的可行性。代码、提示与视频见:https://github.com/sygkounas/MEMENTO。
原文摘要 · Abstract (English)
Long-horizon embodied tasks require policies that execute many dependent actions before task success can be observed. Representing policies as executable control pro- grams (code-as-policy) enables their decision logic to be inspected and revised after rollout evaluation. Revised programs can then be executed and compared by rollout performance, framing policy improvement as execution-guided program search. Evo- lutionary methods driven by large language models (LLMs) provide a natural mecha- nism for this search by generating variants and selecting high-performing candidates. However, existing approaches primarily select among independently generated vari- ants and lack a sequential local improvement phase. We introduce MEMENTO, a memory-guided single-elite memetic framework for code-as-policy evolution. ME- MENTO first evolves a rollout evaluator that maps policy rollouts to scalar fitness and structured feedback metrics. Fitness selects accepted candidates and the next elite, while feedback metrics condition policy proposals generated by memory-guided hill-climbing, macro-mutation, and crossover. We evaluate MEMENTO on two long- horizon embodied domains: Robosuite Franka Tower-of-Hanoi manipulation and AI2- THOR household interaction. MEMENTO outperforms Eureka and REvolve, adapted as code-as-policy evolutionary baselines, in task success and generalization to held- out Robosuite object configurations and unseen AI2-THOR scenes. Ablations show that zero-shot generation and unevolved evaluators fail to solve either domain, and that removing policy-search branches reduces performance. Finally, we deploy the best-evolved Robosuite policy on a physical Franka robot, demonstrating the feasibil- ity of sim-to-real transfer of the evolved code-as-policy. Code, prompts, and videos are available at: https://github.com/sygkounas/MEMENTO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。