用低秩适配器控制批评者更新，提升离线强化学习稳定性
Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning
- 冻结随机初始化主矩阵，只优化低秩适配器
- 训练中批评者损失下降更快，策略性能显著提升
- 适合追求稳定高效的离线强化学习研究者
扩大批评者容量是提升离线强化学习性能的有前途方向。然而，近期研究表明，更大的批评者在基于回放缓冲区的自举训练中容易过拟合和不稳定。本文提出使用低秩适配器（LoRA）作为批评者学习的结构正则化器。方法冻结随机初始化的基矩阵，仅优化对应的低秩适配器，从而将批评者更新限制在低维子空间内。我们在多种离线强化学习算法（包括基于不同网络结构的SAC和FastTD3）上评估该方法。实验表明，LoRA能有效降低训练过程中的批评者损失，提升整体策略性能，在多数任务上达到最佳或具有竞争力的结果。大量实验证明，低秩更新为离线强化学习中的批评者学习提供了一种简单而有效的结构正则化方式。
原文摘要 · Abstract (English)
Scaling critic capacity is a promising direction for improving off-policy reinforcement learning (RL). However, recent work shows that larger critics are prone to overfitting and instability in replay-based bootstrapped training. In this paper, we propose using Low-Rank Adaptation (LoRA) as a structural regularizer for critic learning. Our approach freezes randomly initialized base matrices and optimizes only the corresponding low-rank adapters, thereby constraining critic updates to a low-dimensional subspace. We evaluate our method across different off-policy RL algorithms, including SAC and FastTD3 based on different network architectures. Empirically, LoRA efficiently reduces critic loss during training and improves overall policy performance, achieving the best or competitive results on most tasks. Extensive experiments demonstrate that our low-rank updates provide a simple and effective form of structural regularization for critic learning in off-policy RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。