arXiv:2512.17250cs.AI2025-12被引 1

通过预测与纠错,让多模态大模型游戏控制更流畅

Accelerating Multi-modal LLM Gaming Performance via Input Prediction and Mishit Correction

  • 用世界模型预估下一步动作和状态,提前执行多个动作
  • 新观测到来时,小误差用轻量校正器修正,大误差则重规划
  • 在人体行走任务中降低35%推理次数,延迟减少25%

实时序列控制智能体常受推理延迟制约。即使每步规划延迟微小,也可能导致控制失稳、性能下降。本文提出一种推测-校正框架,将推测执行的‘先预测再验证’思想应用于基于模型的控制(TD-MPC2)。每一步,预训练的世界模型与潜在空间MPC规划器共同生成短时程动作队列及预测的潜在状态轨迹,使智能体可在无需即时重规划的情况下执行多个动作。当新观测到达时,系统测量真实潜在状态与预测状态之间的差异:对于小到中等偏差,使用轻量级学习校正器对推测动作进行残差修正,该修正器离线从重规划教师模型中蒸馏而来;对于大偏差,则安全回退至完整重规划并清空过期动作队列。我们对比了门控双塔MLP校正器与时间注意力变换器校正器,分别应对局部误差与系统漂移。在DMC Humanoid-Walk任务上的实验表明,该方法将规划推理次数从500次降至282次,端到端步骤延迟降低25%,仅带来7.1%的回报下降。消融实验显示,无校正的推测执行在长时程下不可靠,凸显了匹配度感知校正对鲁棒性延迟优化的必要性。

原文摘要 · Abstract (English)

Real-time sequential control agents are often bottlenecked by inference latency. Even modest per-step planning delays can destabilize control and degrade overall performance. We propose a speculation-and-correction framework that adapts the predict-then-verify philosophy of speculative execution to model-based control with TD-MPC2. At each step, a pretrained world model and latent-space MPC planner generate a short-horizon action queue together with predicted latent rollouts, allowing the agent to execute multiple planned actions without immediate replanning. When a new observation arrives, the system measures the mismatch between the encoded real latent state and the queued predicted latent. For small to moderate mismatch, a lightweight learned corrector applies a residual update to the speculative action, distilled offline from a replanning teacher. For large mismatch, the agent safely falls back to full replanning and clears stale action queues. We study both a gated two-tower MLP corrector and a temporal Transformer corrector to address local errors and systematic drift. Experiments on the DMC Humanoid-Walk task show that our method reduces the number of planning inferences from 500 to 282, improves end-to-end step latency by 25 percent, and maintains strong control performance with only a 7.1 percent return reduction. Ablation results demonstrate that speculative execution without correction is unreliable over longer horizons, highlighting the necessity of mismatch-aware correction for robust latency reduction.

多模态控制延迟优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。