Tokenization比多专家机制更能提升深度强化学习性能。
Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL
- 将编码器输出分块为令牌，是SoftMoE有效关键。
- 单个专家配合分块后性能仍优于传统方法。
- 适合关注模型效率与结构设计的研究者。
深度神经网络在强化学习中随模型增大常出现性能下降。尽管软混合专家（SoftMoE）近期在在线强化学习中展现潜力，其有效性原因仍不明确。本文深入分析发现，真正起作用的是对编码器输出进行分块（tokenization），而非多专家结构本身。实验表明，即使使用适当缩放的单个专家，仅通过分块也能保持性能优势，证明了分块是核心机制。
原文摘要 · Abstract (English)
The use of deep neural networks in reinforcement learning (RL) often suffers from performance degradation as model size increases. While soft mixtures of experts (SoftMoEs) have recently shown promise in mitigating this issue for online RL, the reasons behind their effectiveness remain largely unknown. In this work we provide an in-depth analysis identifying the key factors driving this performance gain. We discover the surprising result that tokenizing the encoder output, rather than the use of multiple experts, is what is behind the efficacy of SoftMoEs. Indeed, we demonstrate that even with an appropriately scaled single expert, we are able to maintain the performance gains, largely thanks to tokenization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。