arXiv:2511.08892cs.AI2025-11被引 16

Lumine让AI在3D世界中自主完成数小时任务,像人一样思考与操作。

Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds

  • 用视觉语言模型统一感知、推理与动作,5Hz处理图像生成30Hz操作
  • 在原神中完成5小时主线剧情,效率接近人类水平
  • 零样本跨游戏迁移,无需微调即可玩其他开放世界游戏

我们提出Lumine,首个可在真实挑战性3D开放世界中实时执行数小时复杂任务的通用智能体开放方案。Lumine采用类人交互范式,通过视觉语言模型实现感知、推理与动作的端到端统一,以5Hz处理原始像素,生成30Hz的键盘鼠标指令,并仅在必要时动态调用推理。在《原神》中训练的Lumine成功完成长达五小时的蒙德主线剧情,表现达到人类级效率;同时可理解自然语言指令,在3D开放世界探索与2D GUI操作中完成收集、战斗、解谜和与非玩家角色互动等多样化任务。除本领域性能外,Lumine展现出强零样本跨游戏泛化能力:未经过任何微调,即可完成《鸣潮》100分钟任务及《崩坏:星穹铁道》首章五小时完整剧情。这些成果验证了Lumine在不同虚拟世界与交互模式下的有效性,标志着迈向开放环境通用智能体的重要一步。

原文摘要 · Abstract (English)

We introduce Lumine, the first open recipe for developing generalist agents capable of completing hours-long complex missions in real time within challenging 3D open-world environments. Lumine adopts a human-like interaction paradigm that unifies perception, reasoning, and action in an end-to-end manner, powered by a vision-language model. It processes raw pixels at 5 Hz to produce precise 30 Hz keyboard-mouse actions and adaptively invokes reasoning only when necessary. Trained in Genshin Impact, Lumine successfully completes the entire five-hour Mondstadt main storyline on par with human-level efficiency and follows natural language instructions to perform a broad spectrum of tasks in both 3D open-world exploration and 2D GUI manipulation across collection, combat, puzzle-solving, and NPC interaction. In addition to its in-domain performance, Lumine demonstrates strong zero-shot cross-game generalization. Without any fine-tuning, it accomplishes 100-minute missions in Wuthering Waves and the full five-hour first chapter of Honkai: Star Rail. These promising results highlight Lumine's effectiveness across distinct worlds and interaction dynamics, marking a concrete step toward generalist agents in open-ended environments.

通用智能体3D世界零样本泛化游戏AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。