用动作价值几何优化策略，提升连续控制的采样效率。
SAVGO: Learning State-Action Value Geometry with Cosine Similarity for Continuous Control
- 构建状态-动作联合嵌入空间，使相似价值的动作具有高余弦相似度。
- 在高维任务上优于强基线，显著提升样本效率与性能。
- 适合研究高效强化学习、连续控制方向的学者与工程师。
尽管表示学习和相似性学习已提升强化学习的采样效率，但它们很少直接用于动作空间中的策略更新。为此，本文提出一种几何感知的强化学习算法——状态-动作价值几何优化（SAVGO），显式将基于价值的相似性融入策略更新。SAVGO 在联合状态-动作嵌入空间中学习，使具有相似动作价值估计的样本对呈现高余弦相似度，而差异较大的样本则被映射到不同方向。该学习到的几何结构生成候选动作上的相似性核，使策略优化能直接导向更高价值区域，超越局部梯度更新。因此，表示学习、价值估计与策略优化统一于单一几何一致的目标中，同时保持了离策略演员-评论家训练的可扩展性。方法在标准 MuJoCo 连续控制基准上评估，展示了在高维挑战任务中对强基线的改进。消融实验分析了价值几何学习与基于相似性的策略更新的贡献。
原文摘要 · Abstract (English)
While representation and similarity learning have improved the sample efficiency of Reinforcement Learning (RL), they are rarely used to shape policy updates directly in the action space. To bridge this gap, a geometry-aware RL algorithm that explicitly incorporates value-based similarity into the policy update, State-Action Value Geometry Optimization (SAVGO), is proposed. In detail, SAVGO learns a joint state-action embedding space in which pairs with similar action-value estimates exhibit high cosine similarity, while dissimilar pairs are mapped to distinct directions. This learned geometry enables the generation of a similarity kernel over candidate actions sampled at each update, allowing policy improvement to be guided directly toward higher-value regions beyond local gradient-based updates. As a result, representation learning, value estimation, and policy optimization are unified within a single geometry-consistent objective, while preserving the scalability of off-policy actor-critic training. The proposed method is evaluated on standard MuJoCo continuous-control benchmarks, demonstrating improvements over strong baselines on challenging high-dimensional tasks. Ablation studies are done to analyze the contributions of value-geometry learning and similarity-based policy updates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。