arXiv:2606.24470cs.AI2026-06被引 1

用连续潜空间桥接快慢模型,实现实时游戏智能体的高效决策。

The Latent Bridge: A Continuous Slow-Fast Channel for Real-Time Game Agents

论文配图:The Latent Bridge: A Continuous Slow-Fast Channel for Real-Time Game Agents
图 1 · 摘自论文原文
  • 构建连续潜空间通信通道,避免文本往返延迟。
  • 在MsPacman和RoadRunner上分别提升57%和28%性能。
  • 适合需快速响应但依赖深度规划的实时决策场景。

面向通用计算机应用的实时智能体——以游戏为最严苛场景——必须在几十毫秒内做出动作,同时仍能进行数秒级别的规划。推理型视觉语言模型(Qwen3-VL-8B-Thinking)虽有效但每响应需约1.5秒,远超15Hz控制周期;而反应型模型(MiniCPM-o 4.5)可在毫秒级响应,但在规划任务中表现欠佳。本文将两个规模匹配的冻结模型(9B反应型、8B推理型)耦合,仅训练通信通道。标准文本桥(T)让慢模型输出后缀供快模型读取,而新提出的连续潜空间桥(L)则以LLaVA风格将慢模型残差投影至快模型输入嵌入空间,避免文本往返。在7个Atari游戏和驾驶环境MetaDrive上,针对不同种子微调动作解码器,潜空间桥在所有场景中均不劣于或优于文本桥:在MsPacman和RoadRunner上分别提升57%和28%,其余场景为安全替换。同时使用两通道会引发破坏性干扰(RoadRunner下降96%),故仅推荐单一通道。收益高度可预测:潜空间桥与文本桥增益随慢模型优于快模型的程度线性相关(r=0.93)。MetaDrive为对照负例,因文本桥无增益,潜空间桥亦无作用。论文发布回放记录与可复现流程。

原文摘要 · Abstract (English)

A real-time agent for general computer use - with games as the most demanding case - must act within tens of milliseconds while still planning over seconds. These two regimes sit at opposite ends of the latency-quality tradeoff. A reasoning VLM (Qwen3-VL-8B-Thinking) deliberates effectively but requires ~1.5 s per response - far too slow for a 15 Hz control loop. In contrast, a reactive VLM (MiniCPM-o 4.5) acts in milliseconds but underperforms on planning-heavy tasks. We couple two frozen models of matched scale (9B reactive, 8B reasoning), leaving the communication channel as the sole trainable component. The standard coupling is a Text Bridge (T): the slow model writes a suffix the fast model reads. We introduce a learned continuous Latent Bridge (L) that projects the slow model's residuals into the fast model's input-embedding space in a LLaVA-style manner, avoiding any text round-trip; both are compared against Fast-Only (F). On 7 Atari games and a driving domain (MetaDrive), tuning the action decoder per channel on held-out seeds, the Latent Bridge matches or beats the Text Bridge in every domain: it significantly improves two games (MsPacman +57%, RoadRunner +28%) and is a safe drop-in elsewhere. Combining both channels interferes destructively (RoadRunner -96%), so only one should be used. The benefit is highly predictable: the bridge helps if and only if slow reasoning already beats fast reaction (T > F) - the Latent and Text gains over Fast-Only move together at r=0.93. MetaDrive is the controlled negative, where the Latent Bridge is demonstrably inert because the Text Bridge adds no value. We release replay recordings and reproducible pipelines.

实时决策多模态模型强化学习潜空间

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。