arXiv:2604.13036cs.CV2026-04被引 22

用视频生成3D世界,解决长时间探索时的场景失真问题。

Lyra 2.0: Explorable Generative 3D Worlds

论文配图:Lyra 2.0: Explorable Generative 3D Worlds
图 1 · 摘自论文原文
  • 用3D几何信息路由过去帧,避免记忆丢失
  • 通过自增强历史训练纠正生成误差
  • 适合需要长期探索的虚拟场景构建

近期视频生成技术催生了新范式:通过生成可控制摄像机的视频模拟场景漫游,再用前向重建技术转化为3D模型。该方法结合视频模型的视觉保真度与创意能力,输出可实时渲染和仿真的3D内容。但扩展至大规模复杂环境时,需在长轨迹、大视角变化和重复访问位置下保持3D一致性,现有视频模型在此情境下性能急剧下降。现有方法受限于两种退化:空间遗忘与时间漂移。随着探索进行,先前观察区域超出模型时间上下文,导致重访时产生幻觉;而自回归生成累积微小合成误差,逐渐扭曲场景外观与几何结构。我们提出Lyra 2.0,实现可持久探索的大规模可交互3D世界生成。为解决空间遗忘,我们保留每帧3D几何结构,仅用于信息路由——检索相关历史帧并建立目标视角的密集对应关系,而外观合成仍依赖生成先验。为缓解时间漂移,我们采用自增强历史训练,让模型暴露于自身退化输出,学会纠正漂移而非延续错误。二者结合显著延长且维持3D一致性的视频轨迹,进而微调前向重建模型,实现高质量3D场景的可靠恢复。

原文摘要 · Abstract (English)

Recent advances in video generation enable a new paradigm for 3D scene creation: generating camera-controlled videos that simulate scene walkthroughs, then lifting them to 3D via feed-forward reconstruction techniques. This generative reconstruction approach combines the visual fidelity and creative capacity of video models with 3D outputs ready for real-time rendering and simulation. Scaling to large, complex environments requires 3D-consistent video generation over long camera trajectories with large viewpoint changes and location revisits, a setting where current video models degrade quickly. Existing methods for long-horizon generation are fundamentally limited by two forms of degradation: spatial forgetting and temporal drifting. As exploration proceeds, previously observed regions fall outside the model's temporal context, forcing the model to hallucinate structures when revisited. Meanwhile, autoregressive generation accumulates small synthesis errors over time, gradually distorting scene appearance and geometry. We present Lyra 2.0, a framework for generating persistent, explorable 3D worlds at scale. To address spatial forgetting, we maintain per-frame 3D geometry and use it solely for information routing -- retrieving relevant past frames and establishing dense correspondences with the target viewpoints -- while relying on the generative prior for appearance synthesis. To address temporal drifting, we train with self-augmented histories that expose the model to its own degraded outputs, teaching it to correct drift rather than propagate it. Together, these enable substantially longer and 3D-consistent video trajectories, which we leverage to fine-tune feed-forward reconstruction models that reliably recover high-quality 3D scenes.

3D生成视频生成场景重建一致性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。