解决视觉语言动作模型的时序短视与推理断层问题
MIRTH: Mutual-Information Reasoning with Temporal Hubs for Vision-Language-Action Agents
- 用双尺度时序记忆压缩长期场景与短期运动趋势
- 通过互信息优化推理令牌，实现指令与动作对齐
- 并行解码提升控制吞吐量，适合真实机器人部署
视觉-语言-动作（VLA）模型已成为将网络规模数据中的语义知识迁移至物理机器人控制的强大范式。然而，现有单帧架构存在固有局限：因忽略历史动态导致时序短视、高层指令与低层电机命令间存在推理断层，以及自回归标量解码带来的推理效率低下。本文提出MIRTH，一种统一框架，通过三项创新改进：(1) 双尺度时序记忆枢纽，将长期场景演化与短期运动趋势压缩为紧凑嵌入；(2) 通过互信息目标优化的潜在推理令牌，构建语义规划空间，对齐多模态上下文与动作轨迹；(3) 并行动作解码机制，以向量化预测替代自回归生成，最大化控制吞吐量。在LIBERO仿真基准和真实世界LeRobot平台上的大量评估表明，MIRTH达到当前最优性能，并展现出涌现的错误恢复能力。代码与数据集已开源于http://github.com/kiva12138/mirth。
原文摘要 · Abstract (English)
VLA models have emerged as a powerful paradigm for transferring semantic knowledge from web-scale data to physical robotic control. However, current single-frame architectures suffer from intrinsic limitations: temporal myopia that discards historical dynamics, reasoning gaps between high-level instructions and low-level motor commands, and inference inefficiency due to autoregressive scalar decoding. In this work, we propose MIRTH, a unified framework designed to address these challenges. MIRTH augments a pretrained VLA backbone with three key innovations: (1) dual-scale temporal memory hubs that compress long-term scene evolution and short-term motion trends into compact embeddings; (2) latent reasoning tokens optimized via a mutual-information objective carving out a semantic plan space to align multimodal context with action trajectories; and (3) a parallel action decoding scheme that replaces autoregressive generation with vector-wise prediction to maximize control throughput. Extensive evaluations on the LIBERO simulation benchmark and a real-world LeRobot platform demonstrate that MIRTH achieves state-of-the-art performance and exhibiting emergent error recovery capabilities. The codes and collected datasets are released at http://github.com/kiva12138/mirth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。