用新颖性驱动进化策略训练强化学习中的Transformer模型
Utilizing Novelty-based Evolution Strategies to Train Transformers in Reinforcement Learning
- 基于新颖性设计进化策略NS-ES和NSR-ES，用于训练复杂Transformer架构
- NSR-ES在大型模型上表现稳定，性能与前馈模型相当
- 预训练模型未能显著加速训练，需更多迭代才能产生有效智能体
本文实验了OpenAI-ES的新型新颖性驱动变体NS-ES和NSR-ES，评估其在训练复杂强化学习任务中基于Transformer的架构（如Decision Transformers）时的有效性。同时测试了使用预训练模型是否能加速这些大模型的新颖性训练。实验结果参差不齐：NS-ES虽有进展，但需远超当前迭代次数才能产出有意义的智能体；而NSR-ES则表现出良好的可扩展性，在前馈模型与Decision Transformer上的性能差异小，与此前OpenAI-ES的结果一致，表明其可直接应用于更大规模模型。
原文摘要 · Abstract (English)
In this paper, we experiment with novelty-based variants of OpenAI-ES, the NS-ES and NSR-ES algorithms, and evaluate their effectiveness in training complex, transformer-based architectures designed for the problem of reinforcement learning, such as Decision Transformers. We also test if we can accelerate the novelty-based training of these larger models by seeding the training with a pretrained models. The experimental results were mixed. NS-ES showed progress, but it would clearly need many more iterations for it to yield interesting agents. NSR-ES, on the other hand, proved quite capable of being straightforwardly used on larger models, since its performance appears as similar between the feed-forward model and Decision Transformer, as it was for the OpenAI-ES in our previous work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。