提出可处理图文与图结构数据的通用世界模型,支持多任务零样本推理。
Graph World Model
- 用消息传递机制统一处理文本与图结构信息,支持多模态输入。
- 在6个跨领域任务上性能超越或媲美专用模型,具备多跳推理能力。
- 适合需要跨模态、跨任务泛化的研究者,如机器人规划、智能推荐等。
世界模型在预测、生成和规划任务中表现出强大能力。现有世界模型主要关注非结构化数据,难以利用数字世界中普遍存在的图结构数据。尽管已有多种图基础模型,但它们专注于图学习任务,无法扩展到多模态和跨学科任务。为此,我们提出图世界模型(GWM),一种同时支持非结构化与图结构状态、融合多模态信息,并将多样化任务表示为动作的世界模型。GWM的核心是一个通用的消息传递算法:通过将多模态数据转换为文本(GWM-T)或使用模态特定编码器映射到统一嵌入空间(GWM-E),实现结构化信息聚合。特别地,GWM引入动作节点,通过直接引用或相似性计算连接其他节点,以支持多样任务。在六个不同领域的任务上——包括多模态生成与匹配、推荐、图预测、多智能体、检索增强生成以及规划与优化——实验表明,同一GWM模型的表现优于或媲美领域专用基线,受益于多跳结构,并在未见新任务上展现出强大的零样本/少样本能力。代码已开源:https://github.com/ulab-uiuc/GWM。
原文摘要 · Abstract (English)
World models (WMs) demonstrate strong capabilities in prediction, generation, and planning tasks. Existing WMs primarily focus on unstructured data and cannot leverage the ubiquitous structured data, often represented as graphs, in the digital world. While multiple graph foundation models have been proposed, they focus on graph learning tasks and cannot extend to diverse multi-modal data and interdisciplinary tasks. To address these challenges, we propose the Graph World Model (GWM), a world model that supports both unstructured and graph-structured states with multi-modal information and represents diverse tasks as actions. The core of a GWM is a generic message-passing algorithm to aggregate structured information, either over a unified multi-modal token space by converting multi-modal data into text (GWM-T) or a unified multi-modal embedding space by modality-specific encoders (GWM-E). Notably, GWM introduces action nodes to support diverse tasks, where action nodes are linked to other nodes via direct reference or similarity computation. Extensive experiments on six tasks from diverse domains, including multi-modal generation and matching, recommendation, graph prediction, multi-agent, retrieval-augmented generation, and planning and optimization, show that the same GWM outperforms or matches domain-specific baselines' performance, benefits from multi-hop structures, and demonstrates strong zero-shot/few-shot capabilities on unseen new tasks. Our code for GWM is released at https://github.com/ulab-uiuc/GWM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。