用单纯形嵌入提升强化学习采样效率，不增加训练时间
Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents
- 引入单纯形嵌入，通过几何先验约束表示结构
- 在多种环境上提升采样效率与最终性能
- 适合关注样本效率的强化学习研究者
近期工作通过大规模环境并行化加速了演员-评论家方法的训练速度，但有时仍需大量环境交互才能达到理想性能。鉴于良好结构的表示能提升深度强化学习代理的泛化能力和采样效率，我们提出使用单纯形嵌入：一种轻量级表示层，将嵌入限制在单纯形结构中。这种几何归纳偏置产生稀疏且离散的特征，稳定了评论家自举过程，并增强了策略梯度。在FastTD3、FastSAC和PPO中应用后，单纯形嵌入在多种连续控制和离散控制环境中均一致提升了采样效率与最终性能，且未牺牲运行时速度。
原文摘要 · Abstract (English)
Recent works have proposed accelerating the wall-clock training time of actor-critic methods via the use of large-scale environment parallelization; unfortunately, these can sometimes still require large number of environment interactions to achieve a desired level of performance. Noting that well-structured representations can improve the generalization and sample efficiency of deep reinforcement learning (RL) agents, we propose the use of simplicial embeddings: lightweight representation layers that constrain embeddings to simplicial structures. This geometric inductive bias results in sparse and discrete features that stabilize critic bootstrapping and strengthen policy gradients. When applied to FastTD3, FastSAC, and PPO, simplicial embeddings consistently improve sample efficiency and final performance across a variety of continuous- and discrete-control environments, without any loss in runtime speed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。