构建视觉干扰下的控制表征学习基准,测试模型对背景变化的鲁棒性。
DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractors
- 在深度思维控制套件中设计含动静干扰的多任务数据集
- 发现预训练表征在像素输入下无助于策略学习,存在显著表征差距
- 少量专家数据时,次优数据和随机隐藏目标任务可提升学习效果
通过行为克隆或离线强化学习从历史数据中学习,是避免昂贵在线训练、实现通用智能体规模化的重要方法。然而,现有方法在面对背景、视角等控制无关因素的微小视觉变化时仍表现脆弱。本文提出DeepMind控制视觉基准(DMC-VB),基于DeepMind控制套件构建新数据集,用于评估离线强化学习智能体在存在视觉干扰时,从视觉输入解决连续控制任务的鲁棒性。该数据集相比以往工作具备六大特点:(a)包含不同难度的运动与导航任务;(b)涵盖静态与动态视觉变化;(c)包含不同技能水平策略生成的数据;(d)系统性提供状态与像素观测对;(e)规模大一个数量级;(f)包含隐藏目标任务。我们还提出三个基准以评估表征学习方法的预训练效果,并在多个近期方法上进行实验。结果表明,预训练表征在DMC-VB上无法提升策略学习性能,且基于像素观测与状态学习的策略间存在巨大表征差距。当专家数据有限时,使用次优数据或具有随机隐藏目标的任务进行预训练反而有助于策略学习。数据集与基准代码已开源:https://github.com/google-deepmind/dmc_vision_benchmark。
原文摘要 · Abstract (English)
Learning from previously collected data via behavioral cloning or offline reinforcement learning (RL) is a powerful recipe for scaling generalist agents by avoiding the need for expensive online learning. Despite strong generalization in some respects, agents are often remarkably brittle to minor visual variations in control-irrelevant factors such as the background or camera viewpoint. In this paper, we present theDeepMind Control Visual Benchmark (DMC-VB), a dataset collected in the DeepMind Control Suite to evaluate the robustness of offline RL agents for solving continuous control tasks from visual input in the presence of visual distractors. In contrast to prior works, our dataset (a) combines locomotion and navigation tasks of varying difficulties, (b) includes static and dynamic visual variations, (c) considers data generated by policies with different skill levels, (d) systematically returns pairs of state and pixel observation, (e) is an order of magnitude larger, and (f) includes tasks with hidden goals. Accompanying our dataset, we propose three benchmarks to evaluate representation learning methods for pretraining, and carry out experiments on several recently proposed methods. First, we find that pretrained representations do not help policy learning on DMC-VB, and we highlight a large representation gap between policies learned on pixel observations and on states. Second, we demonstrate when expert data is limited, policy learning can benefit from representations pretrained on (a) suboptimal data, and (b) tasks with stochastic hidden goals. Our dataset and benchmark code to train and evaluate agents are available at: https://github.com/google-deepmind/dmc_vision_benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。