用1比特量化语言模型实现边缘设备上的强化学习,节省资源且性能损失小。
BitRL: Reinforcement Learning with 1-bit Quantized Language Models for Resource-Constrained Edge Deployment

- 采用1比特量化语言模型,权重为-1/0/+1,大幅压缩内存和能耗。
- 在多个基准上保持85%-98%的原模型性能,内存降低10-16倍,能效提升3-5倍。
- 适合资源受限的边缘设备部署,尤其适用于对延迟和隐私要求高的场景。
在资源受限的边缘设备上部署智能强化学习(RL)代理仍面临重大挑战,因为现代深度学习系统需要大量内存、计算和能源。尽管大语言模型(LLMs)已成为强大的决策代理架构,但其数十亿参数规模使其仅能部署于云端,引发延迟、隐私和连接依赖等问题。我们提出BitRL框架,通过使用1比特量化语言模型构建RL代理,实现在严苛资源约束下的本地学习与推理。基于BitNet b1.58架构(三值权重:-1, 0, +1)及优化的推理栈,BitRL相较全精度基线实现10-16倍内存缩减和3-5倍能效提升,同时在各类基准测试中保持85%-98%的任务性能。我们对量化作为结构化参数扰动进行了理论分析,推导出冻结主干架构下量化策略梯度的收敛边界,并揭示了极端量化中的探索-稳定性权衡。该框架系统整合1比特量化语言模型与强化学习,适用于主流硬件部署。
原文摘要 · Abstract (English)
The deployment of intelligent reinforcement learning (RL) agents on resource-constrained edge devices remains a fundamental challenge due to the substantial memory, computational, and energy requirements of modern deep learning systems. While large language models (LLMs) have emerged as powerful architectures for decision-making agents, their multi-billion parameter scale confines them to cloud-based deployment, raising concerns about latency, privacy, and connectivity dependence. We introduce BitRL, a framework for building RL agents using 1-bit quantized language models that enables practical on-device learning and inference under severe resource constraints. Leveraging the BitNet b1.58 architecture with ternary weights (-1, 0, +1) and an optimized inference stack, BitRL achieves 10-16x memory reduction and 3-5x energy efficiency improvements over full-precision baselines while maintaining 85-98 percent of task performance across benchmarks. We provide theoretical analysis of quantization as structured parameter perturbation, derive convergence bounds for quantized policy gradients under frozen-backbone architectures, and identify the exploration-stability trade-off in extreme quantization. Our framework systematically integrates 1-bit quantized language models with reinforcement learning for edge deployment and demonstrates effectiveness on commodity hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。