让四足机器人实时响应指令，不降性能还提速65%
QUART-Online: Latency-Free Large Multimodal Language Model for Quadruped Robot Learning
- 用动作分块离散化压缩动作空间，保留关键信息
- 实现与控制器同步的实时推理，任务成功率提升65%
- 适合需要低延迟、高响应的机器人控制场景
本文针对四足视觉-语言-动作（QUAR-VLA）任务中多模态大语言模型（MLLM）固有的推理延迟问题展开研究。实验发现，传统参数压缩方法会损害语言基础模型在动作指令微调阶段的性能，因而不适用。为此，提出新型无延迟四足MLLM模型QUART-Online，通过动作分块离散化（ACD）将连续动作空间映射到一组离散代表向量，有效压缩表示空间并保留关键信息。随后对MLLM进行微调，实现视觉、语言与压缩动作在统一语义空间中的融合。实验表明，QUART-Online可与现有MLLM系统协同运行，在与底层控制器同频下实现真正实时推理，显著提升各类任务的成功率65%。
原文摘要 · Abstract (English)
This paper addresses the inherent inference latency challenges associated with deploying multimodal large language models (MLLM) in quadruped vision-language-action (QUAR-VLA) tasks. Our investigation reveals that conventional parameter reduction techniques ultimately impair the performance of the language foundation model during the action instruction tuning phase, making them unsuitable for this purpose. We introduce a novel latency-free quadruped MLLM model, dubbed QUART-Online, designed to enhance inference efficiency without degrading the performance of the language foundation model. By incorporating Action Chunk Discretization (ACD), we compress the original action representation space, mapping continuous action values onto a smaller set of discrete representative vectors while preserving critical information. Subsequently, we fine-tune the MLLM to integrate vision, language, and compressed actions into a unified semantic space. Experimental results demonstrate that QUART-Online operates in tandem with the existing MLLM system, achieving real-time inference in sync with the underlying controller frequency, significantly boosting the success rate across various tasks by 65%. Our project page is https://quart-online.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。