用超向量表示动作，实现高效且鲁棒的策略梯度学习
Vector Symbolic Policy Gradient
- 用单位长度超向量表示离散动作，通过相似度评分
- 训练后动作向量是压缩的核记忆，可复用历史经验
- 高维下对随机扰动鲁棒，适合样本效率要求高的场景
我们提出向量符号策略梯度（VSPG），一种基于离散动作的强化学习方法。每个动作由一个单位范数超向量表示，其得分通过与编码状态的相似度计算。在标准softmax策略梯度近似下，我们证明更新过程等价于优势加权的超向量打包并归一化，因此兼容标准优势估计器。进一步表明，每个训练后的动作超向量是一个固定大小的压缩核记忆，存储了访问过的状态的优势加权核展开，并根据编码器诱导的相似性传递证据。这提供了一种无需增加推理时内存即可支持样本高效学习的机制。对于双极动作记忆，我们证明贪婪动作选择在随机比特翻转下稳定，失败概率随超向量维度指数衰减。VSPG因此将向量符号记忆、对数线性策略梯度与核策略搜索联系起来，并提供了定量的鲁棒性保证。
原文摘要 · Abstract (English)
We answer this question with Vector-Symbolic Policy Gradient (VSPG), a discrete-action actor that represents each action by a unit-norm hypervector and scores it by similarity to the encoded state. Under the standard softmax policy-gradient surrogate, we prove that its update is exactly advantage-weighted hypervector bundling followed by normalization, and therefore supports standard advantage estimators. We further show that each trained action hypervector is a fixed-size compressed kernel memory, storing an advantage-weighted kernel expansion over visited states and transferring evidence according to the encoder-induced similarity. This provides a concrete mechanism that can support sample-efficient learning without increasing inference-time memory. Finally, for bipolar action memories, we prove that greedy action selection is stable under random bit flips, with failure probability decaying exponentially in the hypervector dimension. VSPG thus connects VSA action memories, log-linear policy gradients, and kernel policy search while providing a quantitative robustness guarantee.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。