arXiv:2606.03127cs.RO2026-06被引 5

通过优化提示词,让视觉语言动作模型在部署时自动适应新环境。

TTT-VLA: Test-Time Latent Prompt Optimization for Vision-Language-Action Models

论文配图:TTT-VLA: Test-Time Latent Prompt Optimization for Vision-Language-Action Models
图 1 · 摘自论文原文
  • 用潜在提示词在测试时动态调整模型行为,不修改主模型。
  • 在模拟环境中提升任务成功率,关键在于修正少数错误决策。
  • 适合需要实时适应新场景的机器人控制应用。

基于大规模数据训练的视觉-语言-动作(VLA)模型虽取得显著进展,但在部署阶段仍易受分布偏移影响。现有提示引导方法依赖外部指令,我们提出TTT-VLA,一种基于潜在提示优化(LPO)的测试时训练框架。训练阶段,通过额外代理任务学习潜在提示,为策略提供额外条件信号;测试阶段,仅利用当前环境交互数据,通过代理任务的自监督信号优化提示,不修改策略本身。实验表明,该方法在SimplerEnv上显著提升单体与多体设置下的任务成功率。分析显示,性能提升主要源于纠正少数关键决策,而非全局改变策略行为。结果表明,LPO为基础操作策略的部署期优化提供了有效且实用的路径。

原文摘要 · Abstract (English)

Vision-Language-Action (VLA) models trained on large-scale data have made remarkable progress, but they remain vulnerable to distribution shifts at deployment time. Recent VLA models suggest that prompts can serve as an efficient interface for steering policy behavior, but existing prompt-based steering typically relies on external guidance. This raises a natural question: can test-time training (TTT) for VLA be achieved by optimizing a prompt, so that the steering interface itself can be learned and adapted from interaction? We address this question with TTT-VLA, a test-time training framework based on Latent Prompt Optimization (LPO). During training, the latent prompt is learned with an additional proxy task, providing an extra learned conditioning signal for policy learning. At test time, TTT is performed by collecting interaction data from the current environment and optimizing only the latent prompt on those data using the proxy task's self-supervised signal, without modifying the policy itself. Experiments on SimplerEnv demonstrate that the proposed method consistently improves task success rates in both single- and multi-embodiment settings. Further analysis shows that the gains arise primarily from correcting a small number of critical decisions rather than globally altering policy behavior. These results suggest that LPO provides an effective and practical pathway for deployment-time improvement of foundation manipulation policies.

视觉语言动作提示优化测试时训练机器人控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。