arXiv:2608.07314cs.ROcs.CV2026-08

分离语义与动作模块,分时优化提升视觉语言动作模型的操控能力

TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models

  • 将语义和动作组件分开训练,分别用不同频率更新
  • 在CALVIN和真实任务中均超越现有模型,稳定获得更高奖励
  • 适合需要精准控制的机器人操作场景

视觉-语言-动作(VLA)模型通常通过监督微调(SFT)或在线强化学习(RL)进行下游操控任务适配。SFT易受分布偏移影响,而现有RL方法对所有组件采用统一更新策略,忽略其功能差异。本文提出TEMPO,一种语义-动作解耦、双时间尺度的RL后训练框架。TEMPO冻结预训练的视觉-语言主干以保持通用语义表示,仅对两个特定组件进行强化学习优化:语义投影层和底层动作专家。两者以不同速率更新——语义投影层低频更新以维持潜在动作稳定性,动作专家高频更新以快速吸收在线交互中的控制反馈。该解耦策略防止快速策略更新破坏高层语义表示,同时使动作专家能高效学习。在CALVIN基准和真实世界操控任务上的实验表明,TEMPO持续优于现有先进VLA模型及基线RL方法,在两项真实任务中达到并维持更高评估奖励。

原文摘要 · Abstract (English)

Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning (RL) post-training. SFT is prone to distribution mismatch, and existing RL approaches typically apply a single, uniform update strategy to all model components, ignoring their distinct functional roles. We propose TEMPO, a semantic-action decoupled, two-timescale RL post-training framework for VLA models. TEMPO freezes the pretrained vision-language backbone to preserve general semantic representations, and restricts adaptation to two components with dedicated RL optimization loops: the semantic projection layer and the low-level action expert. We update them at different rates--the semantic projection layer infrequently, to keep the latent action stable, and the action expert frequently, to rapidly incorporate control feedback from online interaction. This decoupling RL fine-tuning strategy prevents fast policy updates from destabilizing high-level semantic representations while still allowing the action expert to learn efficiently from online feedback. Experiments on the CALVIN benchmark and real-world manipulation tasks demonstrate that TEMPO consistently outperforms both pretrained state-of-the-art VLA models and the RL post-training baseline, while reaching and maintaining higher evaluation rewards on two real-world tasks.

强化学习机器人操控多模态模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。