用张量分解提升高维强化学习采样效率,适合样本昂贵的场景。
Tensor-Efficient High-Dimensional Q-learning
- 将Q函数建模为低秩CP张量,利用结构化表示减少参数量
- 结合张量误差与访问频率,实现不确定性驱动的探索策略
- 在相同参数预算下,比深度强化学习和矩阵方法更高效
高维强化学习面临复杂计算与低样本效率的问题,尤其在状态动作空间庞大时遭遇维度灾难。传统Q-learning受限于状态动作对数量随问题规模指数增长。尽管基于神经网络的方法如Deep Q-Networks表现良好,但未显式利用问题结构。许多高维控制任务的值函数具有低秩特性,采用低秩张量分解可实现参数高效的表示。然而,现有张量方法仅关注表示精度,未利用结构进行探索。本文提出张量高效Q-learning(TEQL),将Q函数表示为离散状态动作空间上的低秩CP张量,并利用张量结构实现不确定性感知探索。TEQL引入误差-不确定性引导探索(EUGE),融合张量近似误差与访问次数指导动作选择,并采用频率感知正则化稳定更新。在经典控制任务上,相同参数预算下,实验表明TEQL优于矩阵型低秩方法和深度强化学习基线,在样本效率上显著提升,适用于采样成本高的资源受限场景。
原文摘要 · Abstract (English)
High-dimensional reinforcement learning(RL) faces challenges with complex calculations and low sample efficiency in large state-action spaces. Q-learning algorithms struggle particularly with the curse of dimensionality, where the number of state-action pairs grows exponentially with problem size. While neural network-based approaches like Deep Q-Networks have shown success, they do not explicitly exploit problem structure. Many high-dimensional control tasks exhibit low-rank structure in their value functions, and tensor-based methods using low-rank decomposition offer parameter-efficient representations. However, existing tensor-based Q-learning methods focus on representation fidelity without leveraging this structure for exploration. We propose Tensor-Efficient Q-Learning (TEQL), which represents the Q-function as a low-rank CP tensor over discretized state-action spaces and exploits the tensor structure for uncertainty-aware exploration. TEQL incorporates Error-Uncertainty Guided Exploration (EUGE), which combines tensor approximation error with visit counts to guide action selection, along with frequency-aware regularization to stabilize updates. Under matched parameter budgets, experiments on classic control tasks demonstrate that TEQL outperforms both matrix-based low-rank methods and deep RL baselines in sample efficiency, making it suitable for resource-constrained applications where sampling costs are high.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。