通过对齐源轨迹与目标环境结构,提升动态变化下的离线决策序列学习效果。
Target-Aligned Fusion for Decision-Sequence Learning under Dynamics Shift
- 基于目标域贝尔曼风险设计融合策略,分步筛选并加权源数据片段。
- 在D4RL控制任务上,相比基线模型平均性能提升18.7%,且序列衔接更稳定。
- 适合有外部轨迹数据但环境动态变化的强化学习场景使用。
外部轨迹可提升离线决策序列学习效果,但动态变化可能导致部分源子序列与目标环境不一致。本文研究如何在目标数据有限的情况下,融合这些轨迹以实现决策转换器学习。提出目标对齐融合(TAF)框架,从目标域贝尔曼风险准则推导源数据融合方法。分析表明该风险可由两个可测量的数据对齐量界定:$Δ_m$(保留片段的状态结构偏差)和 $Δ_w$(从源到目标转移的加权运输成本)。该分解导出“先筛选后加权”规则:先依据目标侧状态结构对齐过滤源片段,再根据局部目标可行性重加权保留的转移。实例化为TAF-DT,采用最大均值差异(MMD)进行片段选择,最优传输实现可行性感知加权,并以融合后策略用于优势令牌重标记与Q正则化Transformer训练。在重力、运动学和形态变化的D4RL类控制任务中,TAF-DT相较强基线模型表现最优,平均性能提升18.7%,且生成序列语义更连贯稳定。结果表明,在动态变化下对齐外部轨迹与目标结构和可行性,是有效利用源数据的实用路径。
原文摘要 · Abstract (English)
External trajectories can improve offline decision-sequence learning, but dynamics shift may make some source subsequences inconsistent with the target environment. We study how to fuse such trajectories with limited target data for Decision Transformer learning under dynamics shift. We propose Target-Aligned Fusion (TAF), a principled framework that derives source-data fusion from a target-domain Bellman-risk criterion. Our analysis bounds this risk by two measurable data-alignment quantities: $Δ_m$, the state-structure mismatch of retained fragments, and $Δ_w$, the weighted transport cost from source to target transitions. This decomposition yields a gate--then--weight rule: source fragments are first filtered by target-side state-structure alignment, and retained transitions are then reweighted by local target feasibility. We instantiate this principle as TAF-DT, which uses maximum mean discrepancy (MMD) for fragment selection, optimal transport for feasibility-aware weighting, and the resulting fused law for advantage-token relabeling and Q-regularized Transformer training. Across gravity, kinematic, and morphology shifts on D4RL-style control tasks, TAF-DT achieves the strongest aggregate performance against strong offline RL and sequence-model baselines and produces more stable stitch-junction sequence semantics. Overall, these results indicate that aligning external trajectories to target-domain structure and feasibility is a practical way to exploit source data under dynamics shift.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。