多任务离线强化学习中,共享低秩表示能提升泛化性能。
Generalisation in Multitask Fitted Q-Iteration and Offline Q-learning
- 联合学习共享表示与各任务价值函数,通过最小化贝尔曼误差。
- 总样本数越多,估计精度越高,达到1/√(nT)的收敛速度。
- 新任务复用已有表示,降低下游学习复杂度,适合跨任务迁移。
我们研究多任务离线强化学习场景,其中多个任务共享动作价值函数的低秩表示。学习者仅拥有来自多个相关任务的固定数据集,无法进行在线交互,目标是利用共享结构提升统计效率和泛化能力。本文分析了一种多任务拟合Q迭代方法,通过在离线数据上最小化贝尔曼误差,联合学习共享表示与任务特定的价值函数。在标准可实现性和覆盖性假设下,建立了所学价值函数的有限样本泛化保证。分析明确揭示了跨任务数据池化如何提升估计精度,使误差依赖于总样本数的平方根(1/√(nT)),同时保留由分布偏移带来的对时域和集中系数的依赖。此外,考虑下游离线设置:新任务与上游任务共享相同底层表示。研究发现,复用多任务阶段学到的表示可降低下游学习的有效复杂度,相比从零开始学习更具优势。结果阐明了共享表示在多任务离线Q学习中的作用,为模型无关、基于价值的强化学习中何时及如何利用多任务结构改善泛化提供了理论支持。
原文摘要 · Abstract (English)
We study offline multitask reinforcement learning in settings where multiple tasks share a low-rank representation of their action-value functions. In this regime, a learner is provided with fixed datasets collected from several related tasks, without access to further online interaction, and seeks to exploit shared structure to improve statistical efficiency and generalization. We analyze a multitask variant of fitted Q-iteration that jointly learns a shared representation and task-specific value functions via Bellman error minimization on offline data. Under standard realizability and coverage assumptions commonly used in offline reinforcement learning, we establish finite-sample generalization guarantees for the learned value functions. Our analysis explicitly characterizes how pooling data across tasks improves estimation accuracy, yielding a $1/\sqrt{nT}$ dependence on the total number of samples across tasks, while retaining the usual dependence on the horizon and concentrability coefficients arising from distribution shift. In addition, we consider a downstream offline setting in which a new task shares the same underlying representation as the upstream tasks. We study how reusing the representation learned during the multitask phase affects value estimation for this new task, and show that it can reduce the effective complexity of downstream learning relative to learning from scratch. Together, our results clarify the role of shared representations in multitask offline Q-learning and provide theoretical insight into when and how multitask structure can improve generalization in model-free, value-based reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。