一个模型搞定上千种强化学习任务，表现媲美专用模型。
Towards Scalable Multi-Task Reinforcement Learning with Large Decision Models
- 用统一Transformer架构，通过离线轨迹预测下一步动作
- 在1000+异构环境上表现接近专用模型
- 适合做通用智能体的科研与工程人员参考
大规模序列建模的进步表明，单一模型可从多样化数据分布中学习有用表征。受此启发，我们探究是否可用统一的Transformer策略在大量异构强化学习环境中进行训练。我们提出了LDM-v0，一个在数千个跨领域、多模态环境收集的轨迹上离线训练的大决策模型。LDM-v0是一个多任务、多模态的Transformer策略，以观测、动作、奖励和终止信号的历史为条件，通过监督式下一动作预测进行训练。我们描述了环境基础设施、自动化数据生成流程、模型架构和训练方法，并评估其在多种环境中的表现。结果显示，单一预训练模型在约1000个环境（包括机器人、自动驾驶、库存管理、网络安全、交易和视频游戏）上的表现可媲美独立训练的任务专用基准策略。这些结果证明了使用单一Transformer策略在异构强化学习环境上进行大规模离线预训练的可行性。
原文摘要 · Abstract (English)
Recent progress in large-scale sequence modeling has shown that a single model can learn useful representations across highly diverse data distributions. Inspired by these advances, we investigate whether a unified transformer policy can be trained across large collections of heterogeneous reinforcement learning environments. We introduce LDM-v0, a Large Decision Model trained offline on trajectories collected from thousands of environments spanning multiple domains and modalities. LDM-v0 is a multi-task, multi-modal transformer policy conditioned on histories of observations, actions, rewards, and termination signals, and trained through supervised next-action prediction over offline trajectories. We describe the environment infrastructure, automated data generation pipeline, model architecture, and training methodology used to build LDM-v0, and evaluate its performance across diverse environments. We show that a single pretrained model matches the performance of independently trained task-specific reference policies on approximately 1,000 environments including robotics, autonomous driving, inventory management, cybersecurity, trading, and video games. These results demonstrate the feasibility of large-scale offline pretraining across heterogeneous reinforcement learning environments using a single transformer policy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。