让视觉语言模型实时执行动作块,解决延迟导致的卡顿问题。
Real-Time Execution of Action Chunking Flow Policies
- 边执行当前动作块,边生成下一动作块,实现异步无缝衔接。
- 在12个动态任务中提升吞吐量,延迟下仍能精准完成点火等操作。
- 无需重新训练,适配任意基于扩散或流模型的视觉语言系统。
现代人工智能系统,尤其是与物理世界交互的系统,对实时性能要求越来越高。然而,当前先进通用模型(包括近期的视觉-语言动作模型)存在高延迟问题,成为主要挑战。尽管动作分块技术可保证高频控制任务的时间一致性,但未能完全解决延迟问题,导致在分块边界处出现停顿或非分布外的突变动作。本文提出一种新的推理时算法——实时分块(RTC),可直接应用于任何基于扩散或流模型的视觉-语言动作模型,无需重新训练。RTC在执行当前动作块的同时生成下一个动作块,对确定将被执行的动作进行“冻结”,其余部分通过“补全”生成。为验证该方法,我们引入了一个包含12个高度动态任务的新基准测试集(Kinetix模拟器),并评估了6个具有挑战性的双臂操作真实任务。结果表明,RTC具有高效率、高性能,并对推理延迟表现出独特鲁棒性,显著提升任务吞吐量,在存在显著延迟的情况下仍可在精密任务(如点火)中实现高成功率。
原文摘要 · Abstract (English)
Modern AI systems, especially those interacting with the physical world, increasingly require real-time performance. However, the high latency of state-of-the-art generalist models, including recent vision-language action models (VLAs), poses a significant challenge. While action chunking has enabled temporal consistency in high-frequency control tasks, it does not fully address the latency problem, leading to pauses or out-of-distribution jerky movements at chunk boundaries. This paper presents a novel inference-time algorithm that enables smooth asynchronous execution of action chunking policies. Our method, real-time chunking (RTC), is applicable to any diffusion- or flow-based VLA out of the box with no re-training. It generates the next action chunk while executing the current one, "freezing" actions guaranteed to execute and "inpainting" the rest. To test RTC, we introduce a new benchmark of 12 highly dynamic tasks in the Kinetix simulator, as well as evaluate 6 challenging real-world bimanual manipulation tasks. Results demonstrate that RTC is fast, performant, and uniquely robust to inference delay, significantly improving task throughput and enabling high success rates in precise tasks $\unicode{x2013}$ such as lighting a match $\unicode{x2013}$ even in the presence of significant latency. See https://pi.website/research/real_time_chunking for videos.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。