自动化构建智能体评估基准,提升可诊断与可持续性
Intelligent Automation for Embodied Benchmark Construction: Pipelines, Embodiments, Simulators, and Trends

- 按五阶段流程自动化构建智能体评估体系
- 自动化转移成本至验证、审计与版本管理环节
- 适合研究评估框架与长期维护的团队参考
具身智能已涵盖导航、家务协助、操作、自动驾驶、空中代理及多模态大模型控制。这一扩展使评估基准构建成为可靠评测的核心瓶颈。与静态数据集不同,具身基准将任务定义、环境、机器人数据、示范、标注、度量标准、评估脚本和发布策略整合为统一评估系统。本文通过五阶段构建流程(需求与任务设计、数据获取、清洗标注、基准套件生成与度量定义、评估执行与诊断反馈)综述相关文献。分析了从人工标注到传统自动化、基础模型辅助,再到智能体闭环工作流的演进。对比了人力、数据资产获取、计算与仿真、验证调试、治理维护及返工风险等各环节的定性成本。核心结论:自动化并非简单降低基准成本,而是将成本转向验证、审计、版本控制与长期治理。因此,具身评估的进步不仅依赖更大规模的基准套件,更需具备可诊断、可审计、可负责任更新的构建流程。
原文摘要 · Abstract (English)
Embodied intelligence now spans navigation, household assistance, manipulation, autonomous driving, aerial agents, and multimodal large-model control. This expansion has made benchmark construction a central bottleneck for reliable evaluation. Unlike static datasets, embodied benchmarks combine task specifications, environments, robot data, demonstrations, annotations, metrics, evaluation scripts, and release policies into a single evaluation system. This survey reviews the literature through a five-stage construction pipeline: requirement and task construction, data acquisition, data cleaning and annotation, benchmark suite generation and metric definition, and evaluation execution with diagnostic feedback. For each stage, the survey analyzes the transition from manual curation to traditional automation, foundation-model assistance, and agentic closed-loop workflows. It also compares qualitative construction costs across human labor, data and asset acquisition, compute and simulation, validation and debugging, governance and maintenance, and rework risk. The main conclusion is that automation does not simply reduce benchmark cost. Instead, it often shifts cost toward validation, auditability, version control, and long-term governance. Progress in embodied evaluation will therefore depend not only on larger benchmark suites, but also on construction pipelines that are diagnosable, auditable, and responsibly refreshable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。