通过函数图分析稀疏目标强化学习中的失败模式，发现局部支持可高效诊断失败目标。
Functional Graphs for Predicting and Explaining Goal Failure in Sparse Goal-Conditioned RL
- 构建状态到后继的确定性函数图，揭示策略的吸引子与盆地结构。
- 局部目标支持（LGS）≤0.5时，诊断低成功率目标的精确率高达92.1%。
- 提出四类失败模式分类，帮助理解局部支持不足的深层原因。
稀疏目标条件强化学习中，策略失败常被整体成功率掩盖。本文通过贪心评估诱导的确定性函数图分析训练好的目标条件价值策略：每个目标对应一个状态到唯一后继的映射，将行为分解为吸引子和盆地，揭示了学习策略的局部到全局结构。定义局部目标支持（LGS）——衡量有效邻近状态中其贪心后继为目标的比例。在确定性的稀疏网格世界中，零LGS严格禁止从非目标状态进入目标。实证表明，弱LGS是跨更新规则、课程设置、更大网格和瓶颈几何下目标级失败的强诊断指标：在主8×8 TD设置中，固定规则下LGS ≤0.5的预测精度达0.921，召回0.929，F1为0.925，各类变体表现相近。然而，局部支持不足以保证全局成功：部分受支持目标仍因远距离状态被竞争吸引子捕获或盆地结构破碎而失败。因此引入紧凑的后验分类体系——目标主导、竞争主导、部分/争议性、碎片化，以刻画超出局部支持的残余失败模式。结果表明，稀疏目标强化学习的失败可视为结构化的策略诱导动态，且局部一步策略结构提供了低成本的训练后诊断工具。
原文摘要 · Abstract (English)
Sparse goal-conditioned reinforcement learning can produce policies whose failures are hidden by aggregate success rates. We analyze trained goal-conditioned value policies through the deterministic functional graphs induced by greedy evaluation: for each goal, every state maps to a single successor, decomposing behavior into attractors and basins. This reveals a local-to-global structure in learned policies. We define local goal support (LGS), a one-step statistic measuring the fraction of valid neighboring states whose greedy successor is the goal. In deterministic sparse GridWorlds, zero LGS exactly precludes goal entry from non-goal starts. Empirically, weak LGS is a strong diagnostic of goal-level failure across update rules, curricula, larger grids, and bottleneck geometries: the fixed rule LGS <= 0.5 identifies low-success goals with precision 0.921, recall 0.929, and F1 0.925 in the main 8x8 TD setting, with similar performance across variants. However, local support is not sufficient for global success: some supported goals still fail because distant states are captured by competing attractors or fragmented basin structure. We therefore introduce a compact post-hoc taxonomy of policy-induced graphs -- goal-dominant, competitor-dominated, partial/contested, and fragmented -- to characterize residual failure modes beyond local support. These results show that sparse GCRL failures can be understood as structured policy-induced dynamics, and that local one-step policy structure provides a cheap post-training diagnostic for goal-level failure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。