用自洽先验引导探索,让无搜索象棋网络在单次推理中更精准、更强。
Beyond Search-Imitation: Prior-Directed Exploration for Searchless Chess
- 以自身搜索先验为指引,用前向KL替代熵奖励进行探索
- 在10万题集上解题准确率从93.9%提升至94.9%,四步杀棋率升至81%
- 适合追求战术精度与稳定性的无搜索象棋系统研究者
无搜索象棋网络通过模仿强教师(如Leela Chess Zero的Chessformer)的访问次数实现人类大师级水平,但直接模仿搜索是低效代理。本文采用自对弈强化学习微调,将探索机制替换为面向自身MCTS先验的前向、广覆盖KL散度,使探索聚焦于先验认为有潜力的走法;同时引入由价值头不确定性决定的自适应采样温度,一旦局面明朗则快速收紧。仅约2000步训练后,在10万题集上解题准确率从93.9%提升至94.9%,四步杀棋准确率从77%升至81%,且无搜索强度维持或略高于基线。跨匹配算力测试显示战术准确率与对弈强度可分离:准确率提升仅1个百分点,而等级分围绕基线波动;仅优化谜题的对照组虽取得最大战术提升,但损失约260 Elo。分布分析表明,正则化防止自对弈坍缩至单一路线,新解出的难题正是先验曾保留的近胜之招。前向KL先验在等级分上显著领先,与反向KL锚定方法统计无异,后者虽更集中但丢弃了最难解的路径。
原文摘要 · Abstract (English)
Searchless chess networks reach human master strength from a single forward pass by imitating a stronger teacher: the strongest, Leela Chess Zero's (Lc0) released Chessformer, distills the visit counts of an AlphaZero-style Monte Carlo Tree Search (MCTS). Imitating a search is a poor proxy for playing without one, so we fine-tune for single-pass strength with self-play reinforcement learning (RL). Its exploration is usually supplied by an entropy bonus, the reverse Kullback-Leibler (KL) divergence to uniform. We replace it with a forward, mass-covering KL toward the network's own MCTS prior (prior-directed exploration), so exploration covers the moves the prior judges promising, and pair it with an entropy-adaptive sampling temperature, set by the value head's outcome uncertainty, that sharpens once a position is decided. In about two thousand steps it raises puzzle accuracy from 93.9% to 94.9% on a 100,000-puzzle suite and mate-in-four accuracy from 77% to 81% while holding searchless strength at or slightly above the base. Measuring tactical accuracy and playing strength together across a matched-compute sweep, we find the two dissociate: accuracy gains fall in a one-point band while ratings straddle the base, and a control fine-tuned on puzzles alone posts the study's largest tactical gains while shedding roughly 260 Elo; a better puzzle-solver is not thereby a stronger player. Distribution-level measurements show what anchoring buys: without a regularizer self-play collapses onto a single line of play, and the puzzles newly solved are the near misses whose winning move the prior kept alive. The forward-KL prior tops the rating ladder, statistically tied with a reverse-KL anchor that concentrates twice as hard and drops the hardest solutions the mass-covering prior keeps in support.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。