用流模型和三角松弛重加权，解决离线目标导向强化学习中的子目标坍缩问题。
NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RL
- 用条件归一化流替代高斯策略，实现多模态子目标分布建模
- 三角松弛重加权使子目标选择在随机环境下仍稳定，避免可达性偏差
- 理论保证不发生模式坍缩，适合复杂路径规划任务
层级隐式Q学习（HIQL）仅依据价值函数优势选择子目标，存在乐观偏差和模式坍缩双重缺陷：前者将偶然成功误判为技能表现，后者使多模态子目标分布退化为单一高斯均值，常落在不可达区域。本文提出NFTR（归一化流子目标策略与三角松弛重加权）。通过条件归一化流替代高斯策略，并基于闭式模式平均结果证明，归一化流是基于AWR的子目标选择所需的最小生成类。引入基于架构三角不等式的三角松弛评分，乘法修正AWR权重，降低绕行成本超过平均可达性的子目标权重。该松弛项在确定性马尔可夫决策过程（MDP）中沿测地线消失，且在随机动态下仍为组合性破坏的保守上界。重加权数据回报（RWDR）目标保持了AWR的群体级单调改进特性，并支持三部分次优性分解。二者结合使子目标选择在理论上避免前述高斯坍缩，且在随机动态下保持稳定性。代码开源：https://github.com/erdemtbao/NFTR
原文摘要 · Abstract (English)
Hierarchical Implicit Q-Learning (HIQL), an offline goal-conditioned RL method, selects subgoals by value-function advantages alone. This rule has two coupled failure modes. Optimistic bias treats lucky stochastic outcomes as skillful choices, and mode collapse reduces a multi-modal subgoal distribution to a single Gaussian mean that often falls in unreachable regions. We propose NFTR (Normalizing Flows subgoal policies with Triangle-slack Reweighting). A conditional Normalizing Flow replaces the Gaussian policy, and a closed-form mode-averaging result identifies NFs as the minimal generative class for AWR-based subgoal selection. A triangle slack score, built on the architectural triangle inequality without relying on distance accuracy, multiplicatively corrects the AWR weight to downweight subgoals whose detour cost exceeds average reachability. Triangle-slack vanishes on geodesics in deterministic MDPs and remains a conservative upper bound on composability violation under stochastic dynamics. The RWDR objective preserves AWR's population-level monotonic improvement and admits a three-term suboptimality decomposition. Together, these two ingredients yield subgoal selection that provably avoids the Gaussian collapse described above and remains stable under stochastic dynamics. GitHub page: https://github.com/erdemtbao/NFTR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。