GEM让离线强化学习能选出多样且可控的动作,避免盲目妥协。
GEM: Guided Expectation-Maximization for Behavior-Normalized Candidate Action Selection in Offline RL
- 用高斯混合模型结合优势加权的期望最大化,保持动作多样性
- 在D4RL上表现媲美顶尖方法,支持不同候选动作数量灵活调整
- 适合需要稳定、可解释动作选择的工业级强化学习应用
离线强化学习虽能从固定数据集拟合强大价值函数,但部署可靠性仍依赖动作选择机制。当数据集导致动作分布分支或多重模态时,单模态策略提取会模糊竞争性假设,生成缺乏数据支持的“中间”动作,使决策即使在强评判器下也脆弱。我们提出GEM(引导式期望最大化),一种分析框架,实现多模态且可显式控制的动作选择。GEM通过批判器引导、优势加权的EM式更新训练高斯混合模型(GMM)策略,保留独立成分并把概率质量推向高价值区域,同时学习可计算的GMM行为模型以量化支持度。推理时,GEM进行基于候选集的选择:生成并行候选集,使用保守集成下置信界结合行为归一化支持度重排序动作,其中行为对数似然在每个状态的候选集中标准化,确保跨状态和候选预算的稳定可比控制。实验证明,GEM在D4RL基准上表现竞争力,并提供简单的推理时预算控制(候选数量),可调计算资源与决策质量而无需重新训练。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) can fit strong value functions from fixed datasets, yet reliable deployment still hinges on the action selection interface used to query them. When the dataset induces a branched or multimodal action landscape, unimodal policy extraction can blur competing hypotheses and yield "in-between" actions that are weakly supported by data, making decisions brittle even with a strong critic. We introduce GEM (Guided Expectation-Maximization), an analytical framework that makes action selection both multimodal and explicitly controllable. GEM trains a Gaussian Mixture Model (GMM) actor via critic-guided, advantage-weighted EM-style updates that preserve distinct components while shifting probability mass toward high-value regions, and learns a tractable GMM behavior model to quantify support. During inference, GEM performs candidate-based selection: it generates a parallel candidate set and reranks actions using a conservative ensemble lower-confidence bound together with behavior-normalized support, where the behavior log-likelihood is standardized within each state's candidate set to yield stable, comparable control across states and candidate budgets. Empirically, GEM is competitive across D4RL benchmarks, and offers a simple inference-time budget knob (candidate count) that trades compute for decision quality without retraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。