用大模型自身上下文做机器人任务记忆，实现高效智能决策
PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Control
- 用预训练多模态大模型的上下文作为任务记忆，无需额外记忆模块
- 在RoboMME上达60.83%成功率，比基线高16个百分点
- 支持20Hz控制频率，适合实时机器人应用
多模态大语言模型（MLLM）能整合长期视觉历史，在部分可观测条件下推理，并通过少量示例推断行为。然而，视觉-语言-动作（VLA）模型通常未利用其上下文能力作为任务记忆。现有方法通过专用历史机制弥补此缺口。本文提出PonderPounce，复用MLLM原生因果上下文作为机器人记忆。Ponder（系统2）累积任务观测、示范和先前认知于其上下文，可生成子目标文本与示范推理供内部使用；Pounce（系统1）接收当前观测、指令和本体感知，通过异步接口仅获取最新认知令牌及其年龄。两者端到端联合训练，无需专用记忆模块或独立桥接预训练。优化服务实现50%分位延迟78ms（认知更新）和25ms（动作模型调用），支持20Hz动作播放。在基于基础规模数据的RoboMME上，9B和0.8B版本分别达到60.83%与50.04%，优于FrameSamp+Modul的44.51%和当前观测π_{0.5}的17.93%。使用9倍数据时，达75.54%对比基线57.88%。在RoboCasa-DC上，仅依赖动作监督即达12.5%，高于最强示范条件基线11.6%，若替换认知为学习到的零状态则降至8.6%。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) can integrate long visual histories, reason under partial observability, and infer behavior from a few examples. Yet vision-language-action (VLA) models generally inherit pretrained representations without using this contextual capacity as episode memory. Memory-dependent policies address this gap through purpose-built history mechanisms. PonderPounce instead reuses an MLLM's native causal context as robot memory. Ponder, a System2 MLLM, accumulates episode observations, demonstrations, and prior cognition in its native causal context and can generate subgoal text and demonstration reasoning for internal use. Pounce, a System1 VLA, receives the current observation, instruction, and proprioception directly; through the Ponder--Pounce interface, it asynchronously receives only the newest continuous cognition token and its age. Both are jointly trained end to end without a purpose-built memory module or separate bridge pretraining. Optimized serving achieves p50 latencies of 78ms for cognition refresh and 25ms for action-model invocation, supporting 20Hz action playback. On RoboMME with base-scale training data, PonderPounce reaches 60.83% with 9B and 50.04% with 0.8B under the same Pounce architecture and interface, versus 44.51% for FrameSamp+Modul and 17.93% for the current-observation π_{0.5}. With 9x data, it reaches 75.54% versus 57.88% for FrameSamp+Modul. On RoboCasa-DC, the same interface learns from action supervision alone and reaches 12.5% versus 11.6% for the strongest published demonstration-conditioned baseline, falling to 8.6% when cognition is replaced by a learned null state.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。