用棋谱+引擎反馈训练安全模型,下棋时自动避开致命失误。
Oracle-Guided Soft Shielding for Safe Move Prediction in Chess
- 从棋谱学走法,用引擎评估每步风险,构建双模型决策机制。
- 探索强度翻倍时,失误率仍显著低于其他方法,最高降47%。
- 适合需要高安全性、强对抗的博弈系统,如智能棋手开发。
在高风险场景中,仅依赖模仿学习或强化学习的智能体在探索过程中常出现致命错误。现有强化学习方法在象棋等环境中需数十万轮训练才能收敛,计算成本高;而模仿学习虽样本高效,但在分布偏移下脆弱且缺乏主动避险机制。本文提出一种名为奥丁引导软防护(OGSS)的框架,在模仿学习设定下,通过来自棋局专家的反馈学习概率化安全模型,实现更安全的探索。聚焦于象棋领域,我们训练一个基于历史对局的走法预测模型,并独立训练一个基于Stockfish评估的漏着预测模型,以估算每一步的战术风险。推理时,智能体先生成候选着法,再由漏着模型识别高风险选项,并利用结合走法概率与漏着概率的效用函数,选出性能与安全兼顾的动作。该方法使智能体在保持竞争力的同时大幅降低战术失误。在数百盘对阵强引擎的测试中,相较于动作剪枝、SafeDAgger和基于不确定性的采样等方法,我们的方案在探索率提升数倍的情况下,仍保持更低的漏着率,证明其可在不牺牲战术严谨性前提下支持更广泛探索。
原文摘要 · Abstract (English)
In high stakes environments, agents relying purely on imitation learning or reinforcement learning often struggle to avoid safety-critical errors during exploration. Existing reinforcement learning approaches for environments such as chess require hundreds of thousands of episodes and substantial computational resources to converge. Imitation learning, on the other hand, is more sample efficient but is brittle under distributional shift and lacks mechanisms for proactive risk avoidance. In this work, we propose Oracle-Guided Soft Shielding (OGSS), a simple yet effective framework for safer decision-making, enabling safe exploration by learning a probabilistic safety model from oracle feedback in an imitation learning setting. Focusing on the domain of chess, we train a model to predict strong moves based on past games, and separately learn a blunder prediction model from Stockfish evaluations to estimate the tactical risk of each move. During inference, the agent first generates a set of candidate moves and then uses the blunder model to determine high-risk options, and uses a utility function combining the predicted move likelihood from the policy model and the blunder probability to select actions that strike a balance between performance and safety. This enables the agent to explore and play competitively while significantly reducing the chance of tactical mistakes. Across hundreds of games against a strong chess engine, we compare our approach with other methods in the literature, such as action pruning, SafeDAgger, and uncertainty-based sampling. Our results demonstrate that OGSS variants maintain a lower blunder rate even as the agent's exploration ratio is increased by several folds, highlighting its ability to support broader exploration without compromising tactical soundness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。