用内在平滑性优化让机器人动作更稳定流畅。
SmoothVLA: Aligning Vision-Language-Action Models with Physical Constraints via Intrinsic Smoothness Optimization
- 通过轨迹抖动惩罚实现无需外部反馈的平滑优化
- 在LIBERO上平滑度提升13.8%,泛化性能超越监督微调
- 适合需要稳定物理控制的机器人任务研究
视觉-语言-动作(VLA)模型已成为机器人操作的强大范式。然而,现有微调方法面临稳定性和探索性的两难:监督微调受限于示范质量且泛化能力差,而强化学习虽增强探索性,却常导致不稳定的抖动轨迹,违反物理约束。为此,我们提出SmoothVLA,一种新颖的强化学习微调框架,协同优化任务性能与运动平滑性。核心技术是融合二值稀疏任务奖励与基于轨迹抖动的连续密集奖励的物理感知混合奖励函数,该奖励为内在式,仅需从策略回放中直接计算,无需环境反馈或繁琐奖励工程。结合组相对策略优化(GRPO),SmoothVLA将轨迹平滑性显式作为优化先验,引导模型生成符合物理可行性的稳定控制。在LIBERO基准上的大量实验表明,SmoothVLA在平滑度上比标准RL提升13.8%,且显著优于监督微调的泛化表现。本工作提供了一种通过内在奖励优化实现VLA模型与物理世界约束对齐的可扩展方案。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models have emerged as a powerful paradigm for robotic manipulation. However, existing post-training methods face a dilemma between stability and exploration: Supervised Fine-Tuning (SFT) is constrained by demonstration quality and lacks generalization, whereas Reinforcement Learning (RL) improves exploration but often induces erratic, jittery trajectories that violate physical constraints. To bridge this gap, we propose SmoothVLA, a novel reinforcement learning fine-tuning framework that synergistically optimizes task performance and motion smoothness. The technical core is a physics-informed hybrid reward function that integrates binary sparse task rewards with a continuous dense term derived from trajectory jerk. Crucially, this reward is intrinsic, that computing directly from policy rollouts, without requiring extrinsic environment feedback or laborious reward engineering. Leveraging the Group Relative Policy Optimization (GRPO), SmoothVLA establishes trajectory smoothness as an explicit optimization prior, guiding the model toward physically feasible and stable control. Extensive experiments on the LIBERO benchmark demonstrate that SmoothVLA outperforms standard RL by 13.8\% in smoothness and significantly surpasses SFT in generalization across diverse tasks. Our work offers a scalable approach to aligning VLA models with physical-world constraints through intrinsic reward optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。