用上下文学习构建局部Q函数，提升离线强化学习性能
In-Context Compositional Q-Learning for Offline Reinforcement Learning
- 将Q-learning转为上下文推理，通过线性Transformer动态生成局部Q值
- 在厨房、MuJoCo和Adroit任务上分别提升16.4%、8.8%和6.3%
- 无需子任务标签，适合复杂分解任务的离线强化学习场景
准确估计Q函数是离线强化学习的核心挑战。现有方法通常依赖共享全局Q函数，难以捕捉由多种子任务构成的任务的组合结构。我们提出一种名为上下文组合Q学习（ICQL）的离线强化学习框架，将Q学习建模为上下文推理问题，利用线性Transformer从检索到的轨迹中自适应推断局部Q函数，无需显式子任务标签。理论上，在局部Q函数线性可近似和上下文权重准确推断两个假设下，ICQL实现了有界的Q函数近似误差，并支持近优策略提取。实验表明，ICQL在离线设置下显著提升性能：在Kitchen任务上最高提升16.4%，在MuJoCo和Adroit任务上分别提升8.8%和6.3%。结果凸显了上下文学习在鲁棒且组合式价值估计中的未被充分探索潜力，并确立了ICQL作为离线强化学习中一种原则性强且有效的框架。
原文摘要 · Abstract (English)
Accurate estimation of the Q-function is a central challenge in offline reinforcement learning. However, existing approaches often rely on a shared global Q-function, which is inadequate for capturing the compositional structure of tasks that consist of diverse subtasks. We propose In-context Compositional Q-Learning (ICQL), an offline RL framework that formulates Q-learning as a contextual inference problem and uses linear Transformers to adaptively infer local Q-functions from retrieved transitions without explicit subtask labels. Theoretically, we show that, under two assumptions -- linear approximability of the local Q-function and accurate inference of weights from retrieved context -- ICQL achieves a bounded approximation error for the Q-function and enables near-optimal policy extraction. Empirically, ICQL substantially improves performance in offline settings, achieving gains of up to 16.4% on kitchen tasks and up to 8.8% and 6.3% on MuJoCo and Adroit tasks, respectively. These results highlight the underexplored potential of in-context learning for robust and compositional value estimation and establish ICQL as a principled and effective framework for offline RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。