用双脑架构保留视觉语言模型通用能力，提升机器人任务表现
TwinBrainVLA: Unleashing the Potential of Generalist VLMs for Embodied Tasks via Asymmetric Mixture-of-Transformers
- 左脑冻结通用模型，右脑可训练专精任务，通过异构混合机制融合知识
- 在SimplerEnv和RoboCasa上实现显著性能提升，复杂操作成功率更高
- 适合需要保持通用理解力的机器人智能系统研发者
视觉-语言-动作（VLA）模型的核心理念是利用预训练视觉语言模型（VLM）的强大通用能力实现泛化具身智能。然而，标准机器人微调会破坏预训练特征空间，导致“灾难性遗忘”，削弱我们希望利用的通用视觉理解能力。为有效利用未被破坏的VLM通用能力完成机器人任务，本文提出TwinBrainVLA，其包含两个同构的VLM路径：一个冻结的通用模型（称作“左脑”）和一个可训练的专用模型（称作“右脑”）。该架构采用非对称混合变压器（AsyMoT）机制，使右脑能够动态查询并融合来自左脑的完整语义知识与本体感知状态。融合后的表示用于条件化流匹配动作专家，实现精确的连续控制。在SimplerEnv和RoboCasa基准上的实验表明，通过显式保留通用能力，TwinBrainVLA在复杂操作任务中相较基线模型取得显著性能提升。
原文摘要 · Abstract (English)
The fundamental premise of Vision-Language-Action (VLA) models is to harness the extensive general capabilities of pre-trained Vision-Language Models (VLMs) for generalized embodied intelligence. However, standard robotic fine-tuning inevitably disrupts the pre-trained feature space, leading to "catastrophic forgetting" that compromises the general visual understanding we aim to leverage. To effectively utilize the uncorrupted general capabilities of VLMs for robotic tasks, we propose TwinBrainVLA, which coordinates two isomorphic VLM pathways: a frozen generalist (also called "Left Brain") and a trainable specialist (also called "Right Brain"). Our architecture utilizes a Asymmetric Mixture-of-Transformers (AsyMoT) mechanism, enabling the Right Brain to dynamically query and fuse intact semantic knowledge from the Left Brain with proprioceptive states. This fused representation conditions a flow-matching action expert for precise continuous control. Empirical results on SimplerEnv and RoboCasa benchmarks demonstrate that by explicitly retaining general capabilities, TwinBrainVLA achieves substantial performance gains over baseline models in complex manipulation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。