用低秩适配在强化学习中大幅降低策略库内存占用，性能几乎不变。
Memory-Efficient Policy Libraries with Low-Rank Adaptation in Reinforcement Learning
- 用低秩适配（LoRA）微调多任务机器人策略，仅更新少量参数。
- 相比全量微调，内存减少20到160倍，部署50个策略可节省95%存储空间。
- 适合资源受限的机器人应用，尤其需要部署大量专用策略的场景。
在微调大语言模型时，参数高效微调（PEFT）如低秩适配（LoRA）已成功降低内存与计算开销。本文探索该方法能否应用于机器人与强化学习领域，实现更低内存消耗和更优计算性能。我们聚焦于多任务机器人场景，构建专用策略库，其中内存效率尤为关键。采用近端策略优化（PPO）算法，使用LoRA对基线模型在不同任务上进行微调。结果显示，在特定超参数下，LoRA可使内存使用量比全量微调降低20至160倍。这意味着在部署10至50个专用策略时，存储节省可达90%至95%，足以决定是否能将整个策略库驻留内存，而非依赖交换内存。同时，实验表明，全量微调与LoRA微调在任务成功率上无显著差异。
原文摘要 · Abstract (English)
When fine-tuning Large Language Models (LLMs), there has been success in minimizing both memory usage and computation with Parameter-Efficient Fine-Tuning (PEFT), like Low Rank Adaptation (LoRA). In this article, we have explored whether this approach is transferable to the world of robotics and Reinforcement Learning (RL), allowing learning with reduced memory usage and improved computational performance. Specifically, we focused on a version of multi-task robotics, where a library of specialist policies are created. In such a library memory efficiency is especially important. We used a Proximal Policy Optimization (PPO) algorithm and fine-tuned a baseline model to different tasks using LoRA. Our results demonstrate that, depending on the hyperparameters, LoRA can minimize memory usage by a factor of 20-160 compared to full fine-tuning of all layers. This implies a 90-95% storage saving when deploying a library of many (10-50) specialized policies, which can be the differentiating factor between being able to store the entire library in memory or having to use swap-memory in an applied robotics setting. At the same time, our results indicate that there is no significant difference in the success-rate between full fine-tuning and LoRA fine-tuning for the selected tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。