用扩散模型玩俄罗斯方块,发现约束可行动作能显著提分
Diffusion-MPC in Discrete Domains: Feasibility Constraints, Horizon Effects, and Critic Alignment: Case study with Tetris
- 用掩码机制只采样合法落子序列,避免无效动作干扰
- 去除非法动作后得分提升6.8%,存活率提高5.6%
- 短规划周期比长周期更稳,适合奖励稀疏的场景
本文以俄罗斯方块为案例,研究基于扩散模型的模型预测控制(Diffusion-MPC)在离散组合空间中的应用。规划器使用类似MaskGIT的离散去噪器采样候选落子序列,并通过重排序选择动作。分析了三个关键因素:(1) 通过有效动作的逻辑掩码实现可行性约束采样;(2) 使用启发式评分、预训练DQN评价器及混合策略进行重排序;(3) 候选数量与规划时域的计算开销。结果表明,可行性掩码可消除46%的非法动作质量,使得分提升6.8%,存活率提高5.6%;朴素的DQN重排序与回溯质量系统性错位,导致平均决策遗憾达17.6(90%分位36.6)。在稀疏延迟奖励下,较短规划时域优于较长时域,提示长期推演中不确定性累积。整体上,计算配置(候选数K,时域H)决定主要失效模式:小K限制候选质量,大H放大重排序错误与模型不匹配。研究揭示了扩散规划器在离散环境中的结构性挑战,并为评价器整合提供实用诊断依据。
原文摘要 · Abstract (English)
We study diffusion-based model predictive control (Diffusion-MPC) in discrete combinatorial domains using Tetris as a case study. Our planner samples candidate placement sequences with a MaskGIT-style discrete denoiser and selects actions via reranking. We analyze three key factors: (1) feasibility-constrained sampling via logit masking over valid placements, (2) reranking strategies using a heuristic score, a pretrained DQN critic, and a hybrid combination, and (3) compute scaling in candidate count and planning horizon. We find that feasibility masking is necessary in discrete domains, removing invalid action mass (46%) and yielding a 6.8% improvement in score and 5.6% improvement in survival over unconstrained sampling. Naive DQN reranking is systematically misaligned with rollout quality, producing high decision regret (mean 17.6, p90 36.6). Shorter planning horizons outperform longer ones under sparse and delayed rewards, suggesting uncertainty compounding in long imagined rollouts. Overall, compute choices (K, H) determine dominant failure modes: small K limits candidate quality, while larger H amplifies misranking and model mismatch. Our findings highlight structural challenges of diffusion planners in discrete environments and provide practical diagnostics for critic integration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。