提出可生成时序世界模型的Transformer架构，支持灵活上下文长度。
World Machine: Towards Generative World Modeling for Time-Series
- 基于隐状态的Transformer结构，适应不同数据量和上下文
- 在合成数据集上实现传统Transformer无法做到的生成能力
- 适合需要长序列建模与高效推理的时序生成任务
世界模型代表生成式AI的一次范式转变，旨在以结构化且可泛化的形式实现对环境的预测性理解与可控模拟。本文提出World Machine，一种面向时间序列的生成式世界建模架构。该架构基于Transformer，引入隐状态机制，能够适应不同规模的观测数据与上下文。相比传统Transformer在上下文长度增长时计算与内存成本呈平方级上升的问题，该方法显著降低资源消耗。在自建合成数据集Toy1D上的实验验证了方法可行性，展示了传统Transformer不具备的能力，并揭示训练协议中各组件的贡献。
原文摘要 · Abstract (English)
World models represent a paradigm shift in generative AI, pursuing predictive understanding and controllable simulation of environments in a structured and generalizable way. We present World Machine, a generative world-modeling architecture for time series. It is a transformer-based architecture with latent states that enables adaptation to different amounts of observed data and contexts. This shows an improvement over traditional transformers, which have a computational and memory cost that scales quadratically with the context. Experiments on a proposed synthetic dataset, Toy1D, validate the approach's feasibility, demonstrate capabilities not found in conventional transformers, and highlight the contributions of each component of the training protocol.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。