让大模型分步诊断与生成,实现可追踪的程序进化。
REFLEX: Reflective Evolution from LLM Experience

- 将视觉诊断与代码生成分离,由评议员和执行者协同工作
- 在阿克罗博特和秋千任务中少于10次调用就完成求解
- 支持跨轮次知识迁移,适合需要透明性与高效性的强化学习场景
大型多模态语言模型已成为指导演化搜索以获得可解释程序策略的强大工具。然而,现有框架依赖单一模型调用同时完成视觉行为分析与纠错代码生成,导致诊断与修复过程耦合,反馈路径不透明,无法保留跨轮次的算法洞见。为实现可审计且高效的策略搜索,我们主张将视觉诊断与代码生成在结构上解耦。本文提出 REFLEX——一个无需训练的演化框架,实现了该解耦。在 REFLEX 中,一个具备视觉能力的评议员首先将任务相关的行为证据提炼为结构化、可审计的诊断;随后,一个优化文本生成的执行者利用这些诊断及持久自演化的技能记忆库(可复用代码片段)合成子策略。该架构不仅提供透明的变异轨迹,还支持跨轮次程序知识传递。在控制基准(月球着陆器、阿克罗博特、秋千)和36维天线阵列合成任务上的大量评估表明,该方法具有卓越的样本效率。值得注意的是,REFLEX 在阿克罗博特和秋千任务中均在不足10次LLM调用内完成求解,在月球着陆器任务上达到最优归一化加权得分1.092,最终性能极具竞争力,同时显著加速了早期透明策略的发现。
原文摘要 · Abstract (English)
Large multimodal language models (LLMs) have emerged as powerful tools for guiding evolutionary search toward interpretable programmatic policies. However, existing frameworks rely on a monolithic model call to simultaneously interpret visual behavioral evidence and synthesize corrective code. This diagnosis-repair entanglement creates an opaque feedback loop, obscuring the rationale behind mutations and preventing the retention of algorithmic insights across independent runs. To achieve auditable and efficient policy search, we argue that visual diagnosis must be structurally decoupled from code generation. We present REFLEX, a train-free evolutionary framework that operationalizes this decoupling. In REFLEX, a vision-enabled Critic first distills task-specific behavioral evidence into structured, auditable diagnoses. Subsequently, a text-optimized Actor synthesizes child policies using these diagnoses alongside a persistent, self-evolving Skill Memory of reusable code snippets. This architecture not only provides transparent mutation traces but also enables cross-run programmatic knowledge transfer. Extensive evaluations across control benchmarks (Lunar Lander, Acrobot, Pendulum) and a 36-dimensional antenna array synthesis task demonstrate exceptional sample efficiency. Notably, REFLEX solves Acrobot and Pendulum in under 10 LLM calls and reaches a best Normalized Weighted Score of 1.092 on Lunar Lander, achieving highly competitive final performance while significantly accelerating the early-stage discovery of transparent policies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。