将大模型知识压缩到小模型，提升多任务强化学习部署效率
TD-MPC-Opt: Distilling Model-Based Multi-Task Reinforcement Learning Agents
- 用蒸馏技术将317M参数大模型转为1M参数小模型
- 在MT30基准上得分28.45，超越原1M模型的18.93
- 支持低资源环境部署，适合机器人等实际应用
我们提出一种新型基于模型的强化学习知识迁移方法，解决大型世界模型在资源受限环境中的部署难题。该方法将高容量多任务智能体（317M参数）高效蒸馏为紧凑模型（1M参数），在MT30基准上显著提升多种任务性能。蒸馏后模型取得28.45的归一化分数，远超原始1M参数模型的18.93。通过FP16后训练量化进一步压缩模型约50%。该方法缓解了实际部署瓶颈，揭示了大世界模型中的知识表征机制，为机器人等资源受限场景下的高效多任务强化学习系统提供新路径。代码已开源。
原文摘要 · Abstract (English)
We present a novel approach to knowledge transfer in model-based reinforcement learning, addressing the critical challenge of deploying large world models in resource-constrained environments. Our method efficiently distills a high-capacity multi-task agent (317M parameters) into a compact model (1M parameters) on the MT30 benchmark, significantly improving performance across diverse tasks. Our distilled model achieves a state-of-the-art normalized score of 28.45, surpassing the original 1M parameter model score of 18.93. This improvement demonstrates the ability of our distillation technique to capture and consolidate complex multi-task knowledge. We further optimize the distilled model through FP16 post-training quantization, reducing its size by $\sim$50\%. Our approach addresses practical deployment limitations and offers insights into knowledge representation in large world models, paving the way for more efficient and accessible multi-task reinforcement learning systems in robotics and other resource-constrained applications. Code available at https://github.com/dmytro-kuzmenko/td-mpc-opt.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。