让机器人用的多模态大模型动态调整计算量,省电省内存。
DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution
- 根据任务难易自动关闭部分模型计算,减少冗余
- 在CALVIN数据集上降低5.2-6.5倍计算量,内存降2-6倍
- 适合资源受限的机器人部署,兼顾效率与性能
多模态大语言模型(MLLMs)在理解复杂语言与视觉信息方面表现出色,推动了通用机器人智能体的发展。然而,受限于机器人平台的算力和内存,传统MLLM推理需存储数十亿参数并执行海量计算,带来巨大硬件压力。本文提出面向机器人视觉-语言-动作模型(DeeR-VLA)的动态早退出框架(DeeR),通过多出口架构实现按需激活模型规模。模型可根据预设的平均计算成本(如功耗)、峰值计算开销(如延迟)及GPU内存使用情况,自动判断是否提前终止推理,避免无效计算。实验表明,在CALVIN机器人操作基准上,DeeR将大语言模型的计算量降低5.2–6.5倍,显存占用减少2–6倍,且不损失性能。代码与模型已开源。
原文摘要 · Abstract (English)
MLLMs have demonstrated remarkable comprehension and reasoning capabilities with complex language and visual data. These advances have spurred the vision of establishing a generalist robotic MLLM proficient in understanding complex human instructions and accomplishing various embodied tasks. However, developing MLLMs for real-world robots is challenging due to the typically limited computation and memory capacities available on robotic platforms. In contrast, the inference of MLLMs involves storing billions of parameters and performing tremendous computation, imposing significant hardware demands. In our paper, we propose a Dynamic Early-Exit Framework for Robotic Vision-Language-Action Model (DeeR-VLA, or simply DeeR) that automatically adjusts the size of the activated MLLM based on each situation at hand. The approach leverages a multi-exit architecture in MLLMs, which allows the model to terminate processing once a proper size of the model has been activated for a specific situation, thus avoiding further redundant computation. Additionally, we develop novel algorithms that establish early-termination criteria for DeeR, conditioned on predefined demands such as average computational cost (i.e., power consumption), as well as peak computational consumption (i.e., latency) and GPU memory usage. These enhancements ensure that DeeR operates efficiently under varying resource constraints while maintaining competitive performance. On the CALVIN robot manipulation benchmark, DeeR demonstrates significant reductions in computational costs of LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance. Code and checkpoints are available at https://github.com/yueyang130/DeeR-VLA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。