自动生成细粒度测试用例,让机器人任务验证更智能高效
MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Models

- 用多智能体协作自动生成基于自然语言的任务分解与验证逻辑
- 在两个基准上检测故障数与传统方法相当,但能精确定位错误环节
- 适合机器人研发、测试人员快速构建可复用的自动化验证系统
视觉-语言-动作(VLA)模型是融合感知、语言理解与动作生成的新兴机器人控制系统。现有测试方法依赖人工构建的符号化测试用例,需领域知识、成本高,且仅提供任务终态判断,缺乏中间行为分析与故障定位能力。为此,我们提出MANGO框架,通过多智能体协同自动从自然语言描述中生成细粒度测试用例。该框架先构建可复用的原子任务库,再生成模拟器驱动的用例定义,并将复杂指令分解为有序原子动作序列及其对应用例。使用生成、评估、裁判三类智能体通过结构化反馈迭代优化输出。在LIBERO_10和RoboCasa Humanoid Tabletop基准上验证显示,MANGO生成的用例与符号化用例检测故障数量相当,同时实现精准故障定位并提供更丰富的诊断信息。消融实验分析了各组件贡献及初始任务集影响,证明其有效性与鲁棒性。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models are emerging robotic control systems that integrate perception, language understanding, and action generation in a unified architecture. Existing testing approaches for VLA-enabled robots rely on manually constructed symbolic test oracles that determine task success from final environment states. These oracles are costly to construct, require domain expertise, and are often tightly coupled to specific tasks and environments, limiting scalability and reuse. Furthermore, they provide only end-state assessments of task outcomes, offering limited insight into intermediate behavior and fault localization. To address these limitations, we introduce MANGO, a multi-agent framework that automatically generates fine-grained oracles from natural-language descriptions of robotic tasks. MANGO first generates a reusable library of atomic tasks, then generates simulator-grounded oracle definitions for each atomic task, and finally produces executable fine-grained oracles by decomposing complex instructions into ordered sequences of atomic actions and corresponding oracles. The framework uses collaborative Generator, Assessor, and Judge agents that iteratively refine generated artifacts through structured feedback. We evaluate MANGO on the LIBERO_10 and RoboCasa Humanoid Tabletop benchmarks. Results show that MANGO generates executable, fine-grained oracles that detect a similar number of failures as symbolic oracles while accurately localizing them and providing richer diagnostic information. Through ablation studies, we further analyzed component contributions and the effect of initial task set, while preserving oracle quality. Overall, the results show the feasibility and effectiveness of test oracle generation for VLA-enabled robots testing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。