通过自引导流匹配提升视觉语言动作策略，无需外部评判器即可生成高质量动作序列。
Potential-Guided Flow Matching for Vision-Language-Action Policy Improvement
- 用学习的潜在成功轨迹增强每段动作生成，实现无需外部评判器的最佳-𝐾推理。
- 在五项仿真任务中达到最强基线性能，真实世界任务成功率提升，训练算力减少38%。
- 解耦优势加权与潜在训练，防止价值幻觉，适合长时序动作生成场景。
大型视觉-语言-动作（VLA）策略正越来越多地作为动作片段的条件生成模型进行训练。然而部署时会产生混合质量的经验：成功示范、部分完成、可恢复错误和失败，难以用标准模仿学习处理。完整行为克隆（BC）会模仿失败，过滤后BC丢弃有用子轨迹，离线强化学习则需额外大型评判器。我们提出ForesightFlow，一种自引导流匹配策略，在每个生成的动作片段中加入学习的成功潜力轨迹。同一流同时提出并评分候选动作，实现无需外部评判器的最佳-𝐾推理。关键问题是策略改进与价值校准需要不同监督：优势加权应强调高质量动作，但对潜在坐标应用相同权重会抑制失败梯度并导致过度自信评分。为此我们采用解耦优势加权流匹配，仅对动作速度应用指数化优势权重，而统一训练潜在速度。我们进一步推导出一阶边界估计器，支持单次停梯度前向传播计算优势。在五个BEHAVIOR-1K仿真任务和五个真实世界双臂任务中，ForesightFlow优于模仿基线，仿真成功率匹配最强独立评判器基线，真实世界成功率提升，训练算力减少38%。消融实验表明解耦可防止价值幻觉，一阶估计器保持候选排序一致性，自引导采样提升长时序执行效果。
原文摘要 · Abstract (English)
Large vision-language-action (VLA) policies are increasingly trained as conditional generative models over action chunks. Yet deployment produces mixed-quality experience-successful demonstrations, partial completions, recoverable mistakes, and failures-that is difficult to use with standard imitation. Full behavior cloning (BC) imitates failures, filtered BC discards useful sub-trajectories, and offline reinforcement learning adds a large critic. We introduce ForesightFlow, a self-guided flow-matching policy that augments each generated action chunk with a learned success-potential trajectory. The same flow proposes and scores candidate actions, enabling best-of-$K$ inference without an external critic. The key issue is that policy improvement and value calibration require different supervision: advantage weighting should emphasize high-quality actions, but applying the same weights to potential coordinates suppresses failure gradients and creates overconfident scores. We address this with decoupled advantage-weighted flow matching, applying exponentiated advantage weights only to action velocities while training potential velocities uniformly. We further derive a one-step boundary estimator for conditional flow matching, allowing advantage computation with a single stop-gradient forward pass. Across five BEHAVIOR-1K simulation tasks and five real-world bimanual tasks, ForesightFlow improves over imitation baselines, matches the strongest separate-critic baseline in simulation success, improves real-world success, and reduces training compute by $38\%$. Ablations show that decoupling prevents value hallucination, the one-step estimator preserves candidate-ranking fidelity, and self-guided sampling improves long-horizon execution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。