提出两种新型在线强化学习算法，适配设备端微调。
Towards Batch-to-Streaming Deep Reinforcement Learning for Continuous Control
- 设计可与主流批量强化学习兼容的纯在线更新算法
- 在标准任务上性能媲美顶尖在线基线，无需繁琐调参
- 解决从批量到在线迁移时策略性能下降问题，适合真实场景部署
当前先进的深度强化学习方法在连续控制任务中表现优异，但其依赖经验回放、批量更新和目标网络，计算开销大，难以在资源受限设备上运行。新兴的流式深度强化学习通过纯在线更新缓解此问题，在标准基准上取得良好效果。本文提出两种新型流式强化学习算法：流式软演员-评论家（S2AC）和流式确定性演员-评论家（SDAC），专为与先进批量强化学习方法兼容而设计，特别适用于设备端微调场景，如仿真到现实的迁移。两种算法在标准基准上性能接近当前最优流式基线，且无需针对每项环境进行繁琐超参数调优。我们进一步研究了从批量到流式的转换过程，发现简单迁移无法保证预训练策略性能，因此提出一种有原则的解决方案来应对该挑战。
原文摘要 · Abstract (English)
State-of-the-art deep reinforcement learning (RL) methods have achieved remarkable performance in continuous control tasks, yet their computational complexity is often incompatible with the constraints of resource-limited hardware, due to their reliance on replay buffers, batch updates, and target networks. The emerging paradigm of streaming deep RL addresses this limitation through purely online updates, achieving strong empirical performance on standard benchmarks. In this work, we propose two novel streaming deep RL algorithms, Streaming Soft Actor-Critic (S2AC) and Streaming Deterministic Actor-Critic (SDAC), explicitly designed to be compatible with state-of-the-art batch RL methods, making them particularly suitable for on-device finetuning applications such as Sim2Real transfer. Both algorithms achieve performance comparable to state-of-the-art streaming baselines on standard benchmarks without requiring tedious per-environment hyperparameter tuning. We further investigate the batch-to-streaming transition, showing that a naive transition does not guarantee preservation of pre-trained policy performance, and propose a principled approach to address this challenge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。