arXiv:2603.20827cs.RO2026-03

用视觉语言模型自动校准机器鱼仿真参数,直接实现从视频到真实游泳的零样本迁移。

Swim2Real: VLM-Guided System Identification for Sim-to-Real Transfer

  • 通过VLM对比实拍与仿真视频,自动提出参数调整方向。
  • 16个参数同步优化,误差比最优方法低43%,五次实验无异常结果。
  • 适合需要快速部署水下机器人、避免人工调参的研究者。

我们提出Swim2Real,一种无需手动设计搜索流程的校准流水线,利用视觉语言模型(VLM)反馈,从游泳视频中校准16参数机器人鱼仿真器。软体水下机器人校准困难,因非线性流固耦合导致参数空间混沌,简化流体模型引入持续的模拟-现实差距,且可控水下实验难以复现。此前工作需三个手工设计阶段处理此复杂性。VLM比较仿真与真实视频并提出参数更新;回溯线搜索验证每步步长,使接受率从14%提升至42%,通过恢复方向正确但幅度过大的提议。Swim2Real同时校准全部16个参数,在所有电机频率下最接近真实鱼速度(平均绝对误差=7.4毫米/秒),比次优方法低43%,五次运行无异常种子。训练好的策略以50赫兹控制命令转移至实体鱼,完成从游泳视频到真实部署的全流程。下游强化学习策略游得比贝叶斯优化校准的仿真器远12%,比CMA-ES远90%。结果表明,VLM引导校准可直接从视频闭合水下机器人的模拟-现实差距,实现无需人工系统识别的零样本强化学习迁移,迈向自动化通用仿真调优的新一步。

原文摘要 · Abstract (English)

We present Swim2Real, a pipeline that calibrates a 16-parameter robotic fish simulator from swimming videos using vision-language model (VLM) feedback, requiring no hand-designed search stages. Calibrating soft aquatic robots is particularly challenging because nonlinear fluid-structure coupling makes the parameter landscape chaotic, simplified fluid models introduce a persistent sim-to-real gap, and controlled aquatic experiments are difficult to reproduce. Prior work on this platform required three manually tailored stages to handle this complexity. The VLM compares simulated and real videos and proposes parameter updates. A backtracking line search then validates each step size, tripling the accept rate from 14% to 42% by recovering proposals where the direction is correct but the magnitude is too large. Swim2Real calibrates all 16 parameters simultaneously, most closely matching real fish velocities across all motor frequencies (MAE = 7.4 mm/s, 43% lower than the next-best method), with zero outlier seeds across five runs. Motor commands from the trained policy transfer to the physical fish at 50 Hz, completing the pipeline from swimming video to real-world deployment. Downstream RL policies swim 12% farther than those from BayesOpt-calibrated simulators and 90% farther than CMA-ES. These results demonstrate that VLM-guided calibration can close the sim-to-real gap for aquatic robots directly from video, enabling zero-shot RL transfer to physical swimmers without manual system identification, a step toward automated, general-purpose simulator tuning for underwater robotics.

仿真迁移视觉语言模型机器人校准水下机器人

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。