用自然语言直接指导离线元强化学习,实现零样本决策生成。
Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervision
- 通过对比语言与决策嵌入,对齐文本与环境动态语义
- 在MuJoCo和Meta-World上实现高容量零样本泛化
- 适合需要快速适应新任务的智能体开发场景
离线元强化学习通常依赖高质量样本或预探索来推断任务信念,但这些监督信号在未见任务中难以获取。本文提出文本到决策智能体(T2DA),直接从决策任务的原始文本中学习,利用更广泛的语言监督。首先引入广义世界模型,将多任务决策数据编码为动态感知的嵌入空间;随后受CLIP启发,通过对比语言-决策预训练,预测文本描述与决策嵌入的匹配关系,有效弥合语义鸿沟,并使文本嵌入理解环境动态。训练完成后,该智能体可直接根据语言指令生成零样本决策。在MuJoCo和Meta-World基准上的实验表明,T2DA实现了高容量零样本泛化,优于多种基线方法。代码已开源。
原文摘要 · Abstract (English)
Offline meta-RL usually tackles generalization by inferring task beliefs from high-quality samples or warmup explorations. The restricted form limits their generality and usability since these supervision signals are expensive and even infeasible to acquire in advance for unseen tasks. Learning directly from the raw text about decision tasks is a promising alternative to leverage a much broader source of supervision. In the paper, we propose \textbf{T}ext-to-\textbf{D}ecision \textbf{A}gent (\textbf{T2DA}), a simple and scalable framework that supervises offline meta-RL with natural language. We first introduce a generalized world model to encode multi-task decision data into a dynamics-aware embedding space. Then, inspired by CLIP, we predict which textual description goes with which decision embedding, effectively bridging their semantic gap via contrastive language-decision pre-training and aligning the text embeddings to comprehend the environment dynamics. After training the text-conditioned generalist policy, the agent can directly realize zero-shot text-to-decision generation in response to language instructions. Comprehensive experiments on MuJoCo and Meta-World benchmarks show that T2DA facilitates high-capacity zero-shot generalization and outperforms various types of baselines. Our code is available at \textcolor{magenta}{\href{https://github.com/NJU-RL/T2DA}{https://github.com/NJU-RL/T2DA}}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。