提升仿真与现实的关联性,让机器人策略评估更可信。
A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluation

- 系统对比多个仿真平台,分析真实与仿真中策略排名的一致性。
- 发现仿真中策略表现相关性不足,且扰动失败模式差异大。
- 给出仿真调优建议,适合机器人研发与仿真平台设计者参考。
仿真已成为评估和改进视觉-语言-动作(VLA)策略的重要工具,提供可扩展、可复现且可控的替代方案,避免高昂的真实机器人测试成本。尽管近期仿真基准在真实感和多样性上取得显著进展,但这些平台尚未被广泛视为真实世界策略评估的可靠代理。本文从仿真与现实的相关性出发,系统研究了多个仿真平台、VLA策略、任务及扰动因素,评估仿真评价是否能保持真实世界中的策略排序一致性、性能相关性以及扰动下的失败模式。该分析揭示了现有仿真器的局限性,并识别出更贴近真实部署的仿真信号。同时,我们进一步探讨用户如何利用仿真进行策略优化,包括仿真微调何时有效,以及训练后数据量对仿真-现实一致性的影。本工作提出统一框架,用于衡量、解释和提升仿真在VLA策略中的实用性,为仿真器设计者和策略开发实践者提供指导。
原文摘要 · Abstract (English)
Simulation has become an essential tool for evaluating and improving vision-language-action (VLA) policies, offering scalable, reproducible, and controllable alternatives to costly real-world robot evaluation. Recent simulation benchmarks have made substantial progress on realism and diversity, yet these platforms have not been widely adopted as reliable proxies for real-world policy evaluation. In this work, we investigate this issue through the lens of sim-and-real correlation. We conduct a systematic study across multiple simulation platforms, VLA policies, tasks, and perturbation factors, measuring whether simulated evaluation preserves real-world conclusions in terms of policy ranking consistency, performance correlation, and perturbation-wise failure patterns. This analysis allows us to characterize the limitations of existing simulators and identify what kinds of simulation signals are more aligned with real-world deployment. We further examine how users should exploit simulation for policy improvement, including when simulator-based finetuning is beneficial and how the amount of post-training data affects sim-and-real alignment. Overall, our work provides a unified framework for measuring, interpreting, and improving the usefulness of simulation for VLA policies, offering guidance both for simulator designers and for practitioners who use simulation as part of the policy development pipeline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。