用Transformer建模策略权重演化路径，实现无需训练的策略优化
Can We Optimize Deep RL Policy Weights as Trajectory Modeling?
- 将策略权重变化视为轨迹，用自回归Transformer建模学习过程
- 在独立训练实验中收集权重路径数据，训练后可直接推断最优策略
- 适合对高效策略生成、少样本强化学习感兴趣的读者
从随机初始化学习最优策略是深度强化学习的核心问题。随着DRL训练规模增大，将策略网络权重视为新数据模态并探索其潜力变得可行。本文聚焦深度强化学习中的策略学习路径，以历史策略权重的轨迹表示学习过程的演化。受Transformer轨迹建模思想启发，提出TIPL（Transformer as Implicit Policy Learner），采用自回归方式处理策略网络权重。通过运行多个独立的RL训练实验，收集策略学习路径数据，并用于训练TIPL模型。实验表明，TIPL能够拟合策略学习的隐式动态，并仅通过推理即可完成策略网络的优化。
原文摘要 · Abstract (English)
Learning the optimal policy from a random network initialization is the theme of deep Reinforcement Learning (RL). As the scale of DRL training increases, treating DRL policy network weights as a new data modality and exploring the potential becomes appealing and possible. In this work, we focus on the policy learning path in deep RL, represented by the trajectory of network weights of historical policies, which reflects the evolvement of the policy learning process. Taking the idea of trajectory modeling with Transformer, we propose Transformer as Implicit Policy Learner (TIPL), which processes policy network weights in an autoregressive manner. We collect the policy learning path data by running independent RL training trials, with which we then train our TIPL model. In the experiments, we demonstrate that TIPL is able to fit the implicit dynamics of policy learning and perform the optimization of policy network by inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。