arXiv:2501.04879cs.LG2025-01被引 4

用张量低秩分解优化强化学习策略,提升效率与泛化。

Multilinear Tensor Low-Rank Approximation for Policy-Gradient Methods in Reinforcement Learning

  • 将策略参数构造成张量,通过帕拉菲克分解实现低秩约束。
  • 在相同奖励下,计算与样本复杂度显著低于神经网络模型。
  • 适合追求高效策略学习的科研与工程人员参考。

强化学习旨在根据随时间变化的状态选择最优动作,以最大化累积奖励。主流算法分为基于价值和基于策略两类,后者通过参数化概率策略从状态映射到动作。当前多数方法使用神经网络实现策略,但面临收敛性差、架构适配性不足、超参数选择困难及状态-动作表示冗余利用不充分等问题。本文提出采用多线性映射来高效估计策略参数,具体通过帕拉菲克(PARAFAC)分解构建张量低秩策略。核心思想是将策略参数组织为张量,并利用张量补全技术强制低秩结构。我们为多种策略类建立了理论保证,并通过数值实验验证了方法的有效性。结果表明,相较于神经网络模型,该方法在保持相似奖励水平的同时,显著降低计算与样本复杂度。

原文摘要 · Abstract (English)

Reinforcement learning (RL) aims to estimate the action to take given a (time-varying) state, with the goal of maximizing a cumulative reward function. Predominantly, there are two families of algorithms to solve RL problems: value-based and policy-based methods, with the latter designed to learn a probabilistic parametric policy from states to actions. Most contemporary approaches implement this policy using a neural network (NN). However, NNs usually face issues related to convergence, architectural suitability, hyper-parameter selection, and underutilization of the redundancies of the state-action representations (e.g. locally similar states). This paper postulates multi-linear mappings to efficiently estimate the parameters of the RL policy. More precisely, we leverage the PARAFAC decomposition to design tensor low-rank policies. The key idea involves collecting the policy parameters into a tensor and leveraging tensor-completion techniques to enforce low rank. We establish theoretical guarantees of the proposed methods for various policy classes and validate their efficacy through numerical experiments. Specifically, we demonstrate that tensor low-rank policy models reduce computational and sample complexities in comparison to NN models while achieving similar rewards.

强化学习张量分解低秩模型策略梯度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。