让回报值与策略性能对齐，提升强化学习的可控性与泛化能力
Return-to-Go Is More Than a Number: Q-Guided Alignment for Return-Conditioned Supervised Learning
- 用Q值引导条件序列模型，使输入回报值与输出策略表现一致
- 在D4RL基准上实现更优的控制精度和任务完成率
- 适合需要精准回报调控与跨任务泛化的强化学习研究者
条件序列模型（CSMs）通过将回报至目标（RTG）作为控制信号来学习策略。然而，现有方法常将RTG视为简单数值输入，未能将其与策略性能对齐。本文提出Q-ALIGN DT框架，通过确保输出策略的$Q$值与输入RTG一致，实现这一对齐。该方法利用$Q$函数为CSMs提供密集指导，并通过基于RTG扰动的技术对$Q$函数进行微调，确保高RTG始终对应高期望回报的轨迹。理论上，当RTG足够高时，Q-ALIGN DT能高效学习到近似最优策略。实验表明，在D4RL基准上，该方法显著提升了可控性与性能。特别地，模型成功学习到结构化的策略族，能在速度追踪等新任务上有效泛化，而先前方法在此类任务中失效。
原文摘要 · Abstract (English)
Conditioned Sequence Models (CSMs) learn policies by treating return-to-go (RTG) as a control signal. However, existing CSMs often treat the RTGs as simple numerical inputs rather than aligning them with the performance of their policies. In this paper, we propose Q-ALIGN DT, a framework that enforces this alignment by ensuring the $Q$-value of the output policy is consistent with the input RTG. By leveraging a $Q$ function to provide dense guidance to CSMs and further fine-tuning it using an RTG-perturbation technique with the CSM, our method ensures that higher RTGs are consistently mapped to trajectories with higher expected returns. Theoretically, we show that Q-ALIGN DT can efficiently learn the desired policy and output a near-optimal one when the RTG is sufficiently high. Empirically, we demonstrate through extensive experiments that Q-ALIGN DT achieves superior controllability and performance across the D4RL benchmark. Remarkably, our model effectively learns a structured family of policies that maintains precise alignment and generalizes to tasks like velocity-tracking where prior methods fail.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。