用精确扑克牌局验证无模型强化学习在动作遮蔽下的表现
Evaluating Model-Free Policy Optimization in Masked-Action Environments via an Exact Blackjack Oracle
- 构建可精确求解的21点环境，提供真实最优策略作为评估基准
- 无模型算法仅在46%决策单元中匹配最优策略，收益仍大幅偏离理论值
- 强调需用精确基准和负向对照防止误判随机波动为算法优势
无限牌堆赌场21点提供了离散随机控制在动态动作遮蔽下的严格可验证基准。在固定拉斯维加斯规则下（软17、3:2赔付、庄家看牌、任意两张可加倍、分牌后可加倍、最多分四次），通过动态规划推导出覆盖4600个标准决策单元的精确解。该解提供真实动作价值、最优策略标签及理论期望值-0.00161/手。为评估样本效率，三种无模型优化器经模拟交互训练：带逐单元指数移动平均基线的遮蔽REINFORCE、同时扰动随机逼近（SPSA）和交叉熵方法（CEM）。REINFORCE最高效，在10^6手后实现46.37%动作匹配率与-0.04688期望值，优于CEM（39.46%，7.5×10^6次评估）和SPSA（38.63%，4.8×10^6次评估）。但所有方法均存在显著单元条件后悔，表明即便奖励收敛，策略层面仍存严重误差。这说明状态访问稀疏且动态动作遮蔽的表格环境仍具挑战性，而整体奖励曲线可能掩盖关键局部失败。作为负向对照，证明在独立同分布抽牌且不计牌时，最优下注额退化为桌最低限额。更大投注仅提升波动性与破产风险，不改善期望值。这些结果强调需使用精确基准与负向对照，避免将随机波动误认为算法性能。
原文摘要 · Abstract (English)
Infinite-shoe casino blackjack provides a rigorous, exactly verifiable benchmark for discrete stochastic control under dynamically masked actions. Under a fixed Vegas-style ruleset (S17, 3:2 payout, dealer peek, double on any two, double after split, resplit to four), an exact dynamic programming (DP) oracle was derived over 4,600 canonical decision cells. This oracle yielded ground-truth action values, optimal policy labels, and a theoretical expected value (EV) of -0.00161 per hand. To evaluate sample-efficient policy recovery, three model-free optimizers were trained via simulated interaction: masked REINFORCE with a per-cell exponential moving average baseline, simultaneous perturbation stochastic approximation (SPSA), and the cross-entropy method (CEM). REINFORCE was the most sample-efficient, achieving a 46.37% action-match rate and an EV of -0.04688 after 10^6 hands, outperforming CEM (39.46%, 7.5x10^6 evaluations) and SPSA (38.63%, 4.8x10^6 evaluations). However, all methods exhibited substantial cell-conditional regret, indicating persistent policy-level errors despite smooth reward convergence. This gap shows that tabular environments with severe state-visitation sparsity and dynamic action masking remain challenging, while aggregate reward curves can obscure critical local failures. As a negative control, it was proven and empirically confirmed that under i.i.d. draws without counting, optimal bet sizing collapses to the table minimum. In addition, larger wagers strictly increased volatility and ruin without improving expectation. These results highlight the need for exact oracles and negative controls to avoid mistaking stochastic variability for genuine algorithmic performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。