提出DROL方法,让离线强化学习在单步推理下更精准地利用数据支持动作。
Preserve Support, Not Correspondence: Dynamic Routing for Offline Reinforcement Learning

- 用动态路由机制分配数据动作到候选动作,仅更新最匹配的那一个。
- 在OGBench上优于基线FQL,在AntMaze和Adroit任务上保持竞争力。
- 适合追求高效推理且需精准动作选择的离线强化学习场景。
单步离线强化学习策略因其避免反向传播长序列采样器而保持推理高效,但仍需在无偏差批评者指导下不偏离数据可支持的动作。现有单步提取流程中,强迭代教师为每个潜在采样提供单一目标动作,学生输出需同时逼近高Q值并靠近该目标点。当两者方向冲突时,损失会在此样本上妥协,即使附近存在数据支持的更优动作。我们提出DROL,一种基于潜在条件的单步策略,采用top-1动态路由。对每个状态,策略从受限潜先验中采样K个候选动作,将每个数据动作分配给最近的候选动作,并仅以行为克隆和批评者引导更新该胜出候选。由于路由根据当前候选几何结构重新计算,支持区域的归属可在学习过程中动态转移。这使单步策略能实现局部改进,捕捉点式提取难以捕捉的优化,同时保留测试时单次前向推理能力。在OGBench和D4RL上,DROL表现媲美单步FQL基线,提升多个OGBench任务组性能,同时在AntMaze和Adroit上保持强大表现。
原文摘要 · Abstract (English)
One-step offline RL actors are attractive because they avoid backpropagating through long iterative samplers and keep inference cheap, but they still have to improve under a critic without drifting away from actions that the dataset can support. In recent one-step extraction pipelines, a strong iterative teacher provides one target action for each latent draw, and the same student output is asked to do both jobs: move toward higher Q and stay near that paired endpoint. If those two directions disagree, the loss resolves them as a compromise on that same sample, even when a nearby better action remains locally supported by the data. We propose DROL, a latent-conditioned one-step actor trained with top-1 dynamic routing. For each state, the actor samples $K$ candidate actions from a bounded latent prior, assigns each dataset action to its nearest candidate, and updates only that winner with Behavior Cloning and critic guidance. Because the routing is recomputed from the current candidate geometry, ownership of a supported region can shift across candidates over the course of learning. This gives a one-step actor room to make local improvements that pointwise extraction struggles to capture, while retaining single-pass inference at test time. On OGBench and D4RL, DROL is competitive with the one-step FQL baseline, improving many OGBench task groups while remaining strong on both AntMaze and Adroit. Project page: https://muzhancun.github.io/preprints/DROL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。