用知识图谱自动生成可靠多模态任务,提升智能体评估效果。
Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphs
- 基于知识图谱采样子图并结合模板生成任务
- 任务一致性提升20%,可解性提高17%
- 适合评估文档理解与网页交互类智能体
随着多模态大模型驱动的智能体在自主性和泛化能力上不断进步,传统静态数据集面临可扩展性不足的问题,难以全面评估其在复杂多样任务中的表现。现有研究虽尝试用大模型生成任务,但因模型幻觉及缺乏内部数据关系建模,导致任务存在语义不一致和不可解问题。为此,我们提出Graph2Eval——一种基于知识图谱的自动化、可扩展且语义可靠的智能体任务生成框架。该框架利用异构外部数据源构建的知识图谱作为结构化任务空间,通过子图采样与任务模板、元路径策略结合生成多模态任务。为确保任务可靠性,引入基于节点可达性分析、大模型评分与相似性分析的多阶段过滤机制,保障任务多样性与可解性。该框架统一支持RAG Agent与Web Agent场景,高效生成文档理解与多步网页交互任务。我们构建了Graph2Eval-Bench数据集,包含1,319个任务,覆盖两类场景。实验表明,与基线相比,Graph2Eval平均提升任务语义一致性20%、可解性17%,且能有效区分智能体性能,为智能体评估提供新视角。
原文摘要 · Abstract (English)
As multimodal LLM-driven agents advance in autonomy and generalization, traditional static datasets face inherent scalability limitations and are insufficient for fully assessing their capabilities in increasingly complex and diverse tasks. Existing studies have attempted to generate agent tasks using LLMs, but due to the inherent hallucinations of LLMs and the lack of internal data relationship modeling, these tasks often exhibit semantic inconsistencies and solvability issues. To address these challenges, we introduce Graph2Eval, a knowledge-graph-driven framework for automated, scalable, and semantically grounded agent task generation. At its core, Graph2Eval leverages a knowledge graph built from heterogeneous external data sources as a structured task space, generating multimodal agent tasks through subgraph sampling and task construction guided by task templates and meta-path strategies. To further ensure task reliability, a multi-stage filtering pipeline based on node reachability analysis, LLM scoring, and similarity analysis ensures the diversity and solvability of the generated tasks. By unifying both RAG Agent and Web Agent scenarios, Graph2Eval enables efficient generation of multimodal document understanding tasks and multi-step web interaction tasks. We instantiate the framework with Graph2Eval-Bench, a curated dataset of 1,319 tasks spanning document understanding and web interaction scenarios. Extensive experiments show that, on average, Graph2Eval improves task semantic consistency by 20% and solvability by 17% over baselines, while Graph2Eval-Bench effectively distinguishes agent performance, offering a new perspective on agent evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。