用语言描述代替视觉模拟,实现快速安全的智能体决策。
Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Models
- 用语言-动作世界模型替代视觉仿真,通过文本预测动作后果。
- 推理速度提升14倍,任务成功率最高提升15个百分点。
- 适合需要快速响应的安全关键型机器人系统应用。
部署安全关键型智能体需在执行前预判动作后果。尽管世界模型提供了这种前瞻性能力,但依赖视觉仿真的现有方法往往每步延迟超过几秒,开销巨大。本文挑战了视觉处理对故障预防必不可少的假设,证明策略的潜在状态与计划动作已蕴含足够信息以预测结果,使视觉仿真在故障预防中变得冗余。为此,我们提出DILLO(DIstiLLed Language-Action World Model),一种将“先模拟再执行”转变为“先描述再执行”的快速引导层。DILLO通过跨模态蒸馏训练:由具备视觉语言能力的教师模型标注离线轨迹,学生模型则学习基于潜在状态预测语义结果。该方法构建纯文本推理路径,完全绕过繁重的视觉生成,实现比基线14倍的速度提升。在MetaWorld和LIBERO上的实验表明,DILLO能生成高保真度的下一状态描述,并有效引导策略,使任务成功率平均提升9.3个百分点,最高达15个百分点。代码已开源。
原文摘要 · Abstract (English)
Deploying safety-critical agents requires anticipating the consequences of actions before they are executed. While world models offer a paradigm for this proactive foresight, current approaches relying on visual simulation incur prohibitive latencies, often exceeding several seconds per step. In this work, we challenge the assumption that visual processing is necessary for failure prevention. We show that a trained policy's latent state, combined with its planned actions, already encodes sufficient information to anticipate action outcomes, making visual simulation redundant for failure prevention. To this end, we introduce DILLO (DIstiLLed Language-ActiOn World Model), a fast steering layer that shifts the paradigm from "simulate-then-act" to "describe-then-act." DILLO is trained via cross-modal distillation, where a privileged Vision Language Model teacher annotates offline trajectories and a latent-conditioned Large Language Model student learns to predict semantic outcomes. This creates a text-only inference path, bypassing heavy visual generation entirely, achieving a 14x speedup over baselines. Experiments on MetaWorld and LIBERO demonstrate that DILLO produces high-fidelity descriptions of the next state and is able to steer the policy, improving episode success rate by up to 15 pp and 9.3 pp on average across tasks. Code is available at github.com/MaxPappa/DILLO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。