提出低精度概率更新机制,让设备端强化学习更省资源且效果更好。
Contextual Bandits for Resource-Constrained Devices using Probabilistic Learning

- 用随机部分更新+衰减概率替代确定性累加,降低计算开销。
- 3比特下性能接近原始方法,比二值化版本更稳定可靠。
- 适合内存、算力受限的边缘设备部署,如手机或IoT终端。
上下文老虎机(Contextual Bandits, CB)是许多自适应服务背后的在线决策问题。随着对设备端部署的需求增加,必须在严格限制内存、计算和能耗的前提下运行CB智能体。然而,传统线性CB算法因计算与内存成本随维度上升而难以应用。最近提出的基于高维计算(HD-CB)的方法通过进入高维空间来建模和求解CB问题,具有更快收敛速度、良好可扩展性和更高内存效率。但其学习规则依赖累积:动作向量值随时间增长,需高精度表示。虽然周期性二值化可防溢出,却可能丢失幅度信息,影响决策质量。本文提出概率性HD-CB,以概率更新规则替代确定性累积:每步仅随机更新部分向量分量,更新概率随时间衰减,分量值被约束在预定义范围[-k,+k]内。该方法支持低精度实现,无需周期性二值化,避免溢出,且期望更新成本与更新比例成正比。在标准合成数据集上使用Open Bandit Pipeline进行离线评估表明,概率性HD-CB在相同精度下持续优于二值化版本,且仅用3比特/组件即可逼近完整精度的HD-CB性能。
原文摘要 · Abstract (English)
Contextual bandits (CB) are online sequential decision-making problems under partial feedback that underpin many adaptive services. There is a growing demand to deploy CB agents directly on-device, under strict constraints on memory, compute, and energy. However, standard linear CB algorithms are often impractical for resource-constrained devices with their unfavorable scaling in computational and memory costs. Recently, HD-CB, a CB approach based on hyperdimensional computing principles, has been proposed to model and solve CB problems by moving into high-dimensional spaces. HD-CB offers faster convergence, favorable scalability, and improves memory efficiency compared to linear CB algorithms. However, its learning rule is accumulation-based: the values of action vectors grow over time, requiring high precision. While periodic binarization can prevent overflow in low-precision components, it may discard important information about magnitudes and degrade decision quality. This paper introduces probabilistic HD-CB, a low-precision variant that replaces deterministic accumulation with a probabilistic update rule. At each step, only a random subset of vector components is updated, with a time-decaying update probability, and component values are constrained to a predefined range [-k,+k]. This approach enables low-precision components, prevents overflow without periodic binarization, and reduces the expected update cost in proportion to the fraction of updated components. Off-policy evaluation on standardized synthetic CB benchmarks using the Open Bandit Pipeline shows that probabilistic HD-CB consistently outperforms binarized HD-CB at equal precision, while approaching the performance of HD-CB with as few as 3 bits per component.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。