统一评估视觉-语言-动作模型,一键跑通14个仿真基准测试。
vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models
- 通过WebSocket+Docker解耦模型与环境,只需实现一个predict方法
- 支持14个基准、6个模型服务器,2000个任务仅需18分钟完成
- 开源完整评估流程,适合作为研究团队的标准化评测工具
视觉-语言-动作(VLA)模型在多个仿真基准上评估日益增多,但每个基准加入评估流水线都需处理依赖冲突、协议不一致和未文档化的预处理问题,导致评估成本随模型和基准数量指数上升,多数团队难以开展全面评估。我们提出 vla-eval,一个开源评估框架,通过 WebSocket + msgpack 协议与 Docker 环境隔离,将模型推理与基准执行解耦。模型只需实现一个 predict() 方法,基准通过四方法接口接入,即可自动生成完整的交叉评估矩阵。该框架支持14个仿真基准和6个模型服务器。通过任务分片与批量推理,实现最高47倍的时钟速度提升,2000个 LIBERO 任务在约18分钟内完成。为验证框架有效性,我们在六个VLA代码库和三个基准上复现了已发表结果,并记录此前未公开的问题。此外,我们发布一个包含657项已发表结果的VLA排行榜,涵盖17个基准。框架、评估配置及所有复现结果均公开:https://github.com/allenai/vla-evaluation-harness 及 https://allenai.github.io/vla-evaluation-harness/leaderboard。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models are increasingly evaluated across multiple simulation benchmarks, yet adding each benchmark to an evaluation pipeline requires resolving incompatible dependencies, matching underspecified evaluation protocols, and reverse-engineering undocumented preprocessing. This burden scales with the number of models and benchmarks, making comprehensive evaluation impractical for most teams. We present vla-eval, an open-source evaluation harness that eliminates this per-benchmark cost by decoupling model inference from benchmark execution through a WebSocket+msgpack protocol with Docker-based environment isolation. Models integrate once by implementing a single predict() method; benchmarks integrate once via a four-method interface; the full cross-evaluation matrix works automatically. The framework supports 14 simulation benchmarks and six model servers. Parallel evaluation via episode sharding and batch inference achieves up to 47x wall-clock speedup, completing 2,000 LIBERO episodes in ~18 minutes. To validate the framework, we reproduce published scores across six VLA codebases and three benchmarks, documenting previously undocumented pitfalls. We additionally release a VLA leaderboard aggregating 657 published results across 17 benchmarks. Framework, evaluation configs, and all reproduction results are publicly available at https://github.com/allenai/vla-evaluation-harness and https://allenai.github.io/vla-evaluation-harness/leaderboard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。