用表征学习提升多任务强化学习的可扩展性，无需复杂规划
Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
- 用预测性表征+价值函数逼近替代复杂模型规划
- 在多种连续控制任务上超越基线方法，效率更高
- 适合追求高效多任务强化学习的开发者
将强化学习扩展到多样化多任务场景仍是核心挑战。尽管基于模型的强化学习取得显著进展，但其依赖规划和复杂训练流程，难以判断哪些组件真正决定可扩展性。本文重新审视该问题，认为可扩展多任务强化学习的关键并非基于模型的控制，而是表征学习。我们发现，将预测性、基于模型的表征与高容量价值函数近似结合，即可实现优异性能，甚至无需规划。我们提出一种简单无模型算法MR.Q，集成辅助预测目标于可扩展的演员-评论家架构中。该方法在多样化的多任务连续控制任务上优于近期基于世界模型的方法及多种深度强化学习基线，同时显著降低计算开销并提升实际运行效率。我们观察到模型容量增加时性能持续提升，并通过消融实验证明预测表征学习对性能至关重要。
原文摘要 · Abstract (English)
Scaling reinforcement learning (RL) to diverse multitask settings remains a central challenge. While recent advances in model-based RL achieve strong performance, they rely on planning and complex training pipelines, making it unclear which components are essential for scalability. We revisit this question and argue that the primary driver of scalable multitask RL is not model-based control, but \emph{representation learning}. In particular, we show that combining predictive, model-based representations with high-capacity value function approximation is sufficient to achieve strong performance, even without planning. We evaluate a simple model-free algorithm, MR.Q, coupled with auxiliary predictive objectives into a scalable actor-critic architecture. This approach outperforms a recent world-model-based method and a range of deep RL baselines across a diverse suite of multitask continuous control tasks, while significantly reducing computational overhead and improving wall-clock efficiency. We observe consistent improvements with increased model capacity and show through ablations that predictive representation learning is critical for performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。