将失败经验转化为动作级修正信号,提升机器人视觉语言动作策略的鲁棒性。
RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policy

- 通过上下文感知机制识别导致失败的动作并寻找相似情境下的成功替代动作。
- 在真实任务中将成功率从56.7%提升至74.7%,仅需少量训练样本。
- 适合需要高效利用失败数据的机器人操控研究者使用。
流匹配视觉-语言-动作(VLA)策略在机器人操作中展现出巨大潜力,但在部署时常因分布偏移导致误差累积。尽管离线强化学习可利用回放数据改进策略,现有方法或忽略失败数据,或仅在轨迹层面利用,学习效率低且错误持续存在。本文提出RedFlow,一种细粒度的离线强化学习框架,将失败经验转化为流匹配VLA策略的动作级修正监督。RedFlow包含两个关键组件:(1) 上下文感知修正匹配机制,用于识别引发失败的动作,并从相似上下文中检索成功替代动作作为修正目标;(2) 自适应重定向目标,联合增强成功动作、抑制不良动作,并将可恢复的失败引导至修正目标。通过将成功与失败经验均转化为密集监督,RedFlow实现了对混合质量数据的鲁棒恢复学习。在LIBERO基准及三个真实世界操作任务上的实验表明,RedFlow持续优于现有先进离线强化学习基线,将真实世界成功率从56.7%提升至74.7%,且仅需约十分之一的训练样本即可达到强在线方法(PPO、GRPO、DDPO)的效果。
原文摘要 · Abstract (English)
Flow-matching Vision-Language-Action (VLA) policies have shown strong potential for robotic manipulation but often suffer from compounding errors caused by distribution shifts during deployment. While offline reinforcement learning (RL) provides a practical way to improve deployed policies using rollout data, existing methods either ignore failure data or exploit it only at the trajectory level, resulting in low learning efficiency and persistent errors. We propose **RedFlow**, a fine-grained offline RL framework that redirects failure experiences into action-level corrective supervision for flow-matching VLA policies. RedFlow consists of two key components: (1) a **Context-Aware Corrective Matching** mechanism that identifies failure-inducing actions and retrieves successful alternatives from similar contexts as corrective targets, and (2) an **Adaptive Redirection Objective** that jointly reinforces successful actions, suppresses undesirable ones, and redirects recoverable failures toward corrective targets. By converting both successful and failed experiences into dense supervision, RedFlow enables robust recovery learning from mixed-quality data. Experiments on the LIBERO benchmark and three real-world manipulation tasks show that RedFlow consistently outperforms state-of-the-art offline RL baselines, improving the real-world success rate from 56.7% to 74.7%. It also matches strong on-policy methods (PPO, GRPO, and DDPO) while requiring roughly an order of magnitude fewer training samples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。