用低秩张量优化多任务强化学习的值函数，提升数据效率
A Tensor Low-Rank Approximation for Value Functions in Multi-Task Reinforcement Learning
- 将多任务Q函数建模为高阶张量，施加低秩约束以捕捉任务相似性
- 在倒立摆和无线设备场景中，显著降低数据需求并提升训练效率
- 无需预先指定相似任务对，自动从少量数据中推断任务关联
为实现能在物理环境中训练的强化学习系统，本文研究多任务方法以减少对海量数据的需求。在表格化场景下，将跨任务收集的Q函数建模为高阶张量，提出对聚合Q张量施加低秩约束。该方法利用相关任务可能需要相似动作的特性，通过低秩结构隐式强制任务间的相似性，无需事先定义哪些任务相关，而是与张量的随机优化同步从有限数据中推断出这种关系。在两个数值实验中验证了该方法的高效性：首先在由多个倒立摆组成的基准环境，其次在涉及多个无线通信设备的实际场景中，均展现出优异的数据利用效率。
原文摘要 · Abstract (English)
In pursuit of reinforcement learning systems that could train in physical environments, we investigate multi-task approaches as a means to alleviate the need for massive data acquisition. In a tabular scenario where the Q-functions are collected across tasks, we model our learning problem as optimizing a higher order tensor structure. Recognizing that close-related tasks may require similar actions, our proposed method imposes a low-rank condition on this aggregated Q-tensor. The rationale behind this approach to multi-task learning is that the low-rank structure enforces the notion of similarity, without the need to explicitly prescribe which tasks are similar, but inferring this information from a reduced amount of data simultaneously with the stochastic optimization of the Q-tensor. The efficiency of our low-rank tensor approach to multi-task learning is demonstrated in two numerical experiments, first in a benchmark environment formed by a collection of inverted pendulums, and then into a practical scenario involving multiple wireless communication devices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。