arXiv:2508.21046cs.CVcs.RO2025-08NeurIPS被引 74

通过指令路由与稀疏化,让视觉语言动作模型更高效准确

CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsification

  • 用指令引导视觉和语言信息的聚合与剪枝,实现精炼输入
  • 在LIBERO上达97.4%成功率,训练成本降2.5倍,推理快2.8倍
  • 适合机器人任务开发,对算力要求低且性能强

近期基于预训练视觉语言模型的视觉-语言-动作(VLA)模型需大量后训练,计算开销高,限制可扩展性与部署。我们提出CogVLA,一种认知对齐的视觉-语言-动作框架,通过指令驱动路由与稀疏化提升效率与性能。CogVLA受人类多模态协同启发,采用三阶段渐进式架构:1)基于Encoder-FiLM的聚合路由(EFA-Routing)将指令信息注入视觉编码器,选择性聚合并压缩双流视觉标记,形成指令感知的潜在表示;2)基于LLM-FiLM的剪枝路由(LFP-Routing)通过剪除与指令无关的视觉锚定标记,将语言模型中引入动作意图,实现标记级稀疏;3)为确保压缩感知输入仍支持精准连贯的动作生成,引入V-L-A耦合注意力(CAtten),结合因果视觉-语言注意力与双向动作并行解码。在LIBERO基准与真实机器人任务上的实验表明,CogVLA分别取得97.4%和70.0%的成功率,相比OpenVLA训练成本降低2.5倍,推理延迟减少2.8倍。CogVLA已开源,项目地址:https://github.com/JiuTian-VL/CogVLA。

原文摘要 · Abstract (English)

Recent Vision-Language-Action (VLA) models built on pre-trained Vision-Language Models (VLMs) require extensive post-training, resulting in high computational overhead that limits scalability and deployment.We propose CogVLA, a Cognition-Aligned Vision-Language-Action framework that leverages instruction-driven routing and sparsification to improve both efficiency and performance. CogVLA draws inspiration from human multimodal coordination and introduces a 3-stage progressive architecture. 1) Encoder-FiLM based Aggregation Routing (EFA-Routing) injects instruction information into the vision encoder to selectively aggregate and compress dual-stream visual tokens, forming a instruction-aware latent representation. 2) Building upon this compact visual encoding, LLM-FiLM based Pruning Routing (LFP-Routing) introduces action intent into the language model by pruning instruction-irrelevant visually grounded tokens, thereby achieving token-level sparsity. 3) To ensure that compressed perception inputs can still support accurate and coherent action generation, we introduce V-L-A Coupled Attention (CAtten), which combines causal vision-language attention with bidirectional action parallel decoding. Extensive experiments on the LIBERO benchmark and real-world robotic tasks demonstrate that CogVLA achieves state-of-the-art performance with success rates of 97.4% and 70.0%, respectively, while reducing training costs by 2.5-fold and decreasing inference latency by 2.8-fold compared to OpenVLA. CogVLA is open-sourced and publicly available at https://github.com/JiuTian-VL/CogVLA.

视觉语言动作稀疏化机器人高效模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。