统一视觉语言动作模型的记忆与控制，提升长序列任务表现。
UniMem: Unifying Multimodal Memory and Control for Vision-Language-Action Models
- 用事件分类器和关键帧编码统一管理多模态记忆
- 仿真任务成功率93.4%，硬件任务80.0%，显著优于基线
- 适合需要长期记忆的机器人控制、智能体系统应用
尽管视觉-语言-动作（VLA）模型通过互联网规模预训练和任务聚焦微调在长时序任务中表现出色，但在需要记忆的非马尔可夫任务中仍表现不佳。现有方法通常依赖额外的视觉-语言模型（VLM）进行长期记忆管理，造成记忆瓶颈和训练流程断裂。通过固定间隔采样历史帧虽能提供更丰富的过去场景特征，但若采样不恰当会降低性能。为此，我们提出UniMem框架，将高层多模态记忆与底层控制统一于单一主干网络。UniMem采用事件分类器实现记忆更新，关键帧编码器构建密集空间记忆，并引入关键帧缓存技术减少策略推理开销。我们在五个仿真和四个硬件任务上评估了UniMem，涵盖顺序与空间记忆需求，结果表明该统一单模型系统在仿真任务中表现优于固定间隔图像采样基线（93.4% vs. 68.2%），在硬件任务中优于分层基线（80.0% vs. 43.5%），同时具备更快推理速度和更简单的训练流程，便于实际部署。
原文摘要 · Abstract (English)
While Vision-Language-Action (VLA) models have leveraged internet-scale pretraining and task-focused finetuning to achieve strong performance on long-horizon tasks, they often struggle with non-Markovian tasks that require memory. Existing approaches to memory typically involve additional Vision-Language-Models (VLMs) for long-term memory management, introducing a memory bottleneck and a fractured training pipeline. Conditioning on multiple historical frames can provide the VLA with access to more descriptive features of past scenes, but can degrade performance if frames are chosen at arbitrary, fixed intervals. To address these limitations, we present UniMem, a framework that unifies high-level, multimodal memory and low-level control under one backbone. UniMem employs an event classifier for memory updates, a keyframe encoder for dense spatial memory, and a keyframe caching technique to minimize overhead during policy rollouts. We evaluate UniMem across five simulation and four hardware tasks targeting sequential and spatial memory, demonstrating that our unified, single-model system outperforms fixed-interval image sampling baselines (93.4% vs. 68.2%) in simulation and hierarchical baselines (80.0% vs. 43.5%) in hardware, while offering faster inference and a simple training pipeline for easy adoption. Project website: https://losterberg3.github.io/unimem-vla/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。