简单模型参数量增大后性能超越复杂架构，证明规模是关键。
Multi-Task Reinforcement Learning Enables Parameter Scaling
- 用简单基线模型增大规模，直接提升多任务强化学习性能。
- 扩大评判器（critic）参数比扩大执行器（actor）带来更大收益。
- 任务越多越稳定，能有效缓解学习过程中的遗忘问题。
多任务强化学习（MTRL）旨在让单一智能体在多个任务上表现良好。近期研究聚焦于设计复杂新颖的网络结构以提升性能，常导致模型变大；但性能提升究竟源于架构设计还是参数量增加尚不明确。本文通过实验证明，将简单基线模型盲目放大至与复杂架构相当的参数量后，其性能反而更优，且主要得益于对评判器（critic）的扩展。此外，我们发现任务多样性带来的训练稳定性优势：任务数量越多，越能缓解塑性衰减（plasticity loss）。结果表明，多任务同时训练天然支持参数规模的有效扩展，挑战了复杂架构创新的必要性。
原文摘要 · Abstract (English)
Multi-task reinforcement learning (MTRL) aims to endow a single agent with the ability to perform well on multiple tasks. Recent works have focused on developing novel sophisticated architectures to improve performance, often resulting in larger models; it is unclear, however, whether the performance gains are a consequence of the architecture design itself or the extra parameters. We argue that gains are mostly due to scale by demonstrating that naively scaling up a simple MTRL baseline to match parameter counts outperforms the more sophisticated architectures, and these gains benefit most from scaling the critic over the actor. Additionally, we explore the training stability advantages that come with task diversity, demonstrating that increasing the number of tasks can help mitigate plasticity loss. Our findings suggest that MTRL's simultaneous training across multiple tasks provides a natural framework for beneficial parameter scaling in reinforcement learning, challenging the need for complex architectural innovations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。