AlphaZero的搜索策略会真正内化为网络能力,而非临时借用。
When Search Teaches Style: Causal Internalization of Tactical Priors in AlphaZero
- 通过切断搜索引导,检验网络是否真正吸收战术先验
- 无搜索时仍保留33.8%防御响应,证明行为已内化
- 适合研究强化学习中搜索与网络协同机制的学者
AlphaZero通常被视为一个融合了策略-价值网络与蒙特卡洛树搜索的单一智能体。这种融合掩盖了一个因果问题:当自对弈搜索获得有用先验时,网络是否会吸收该行为,还是仅在测试时临时依赖搜索?本文提出跨阶段先验干预(CPI),在训练和评估阶段独立开关根节点战术先验,分离先验的在线作用与网络权重中残留的学习效应。目标明确:在9×9五子棋和19×19围棋的一次性封闭测试中,移除先验后网络仍能维持26.3%(五子棋)和33.8%(围棋)的防御响应,远高于初始水平(分别为13.8%和0.6%),且软重加权效果等同于硬性动作剪枝,说明非动作修剪是机制。重新启用先验可恢复近100%响应,依赖缺口分别为73.7和65.8分。位置锁定评估表明,残余效应源于训练中形成的几何结构,在分布内防御上提升17.3分,但在结构新颖场景下仅+1.3分。因此,搜索更应被视为训练期的行为课程,其成果真实、部分且受几何限制;在线表现与内化能力是不同估计量,常规对角消融无法区分。
原文摘要 · Abstract (English)
AlphaZero is normally evaluated as one agent: a policy-value network fused with Monte Carlo tree search. That fusion hides a causal question. When self-play search is given a useful prior, does the network absorb the induced behavior, or does the behavior stay rented from search at test time? We answer with Cross-Phase Prior Intervention (CPI), which switches a root-level tactical prior on and off independently during training and during evaluation, separating the prior's online effect from the learned residual it leaves in the weights. The endpoint is deliberately narrow: how often a network discharges a forced defensive obligation when no search-time guidance is available. On a sealed one-shot final test in 9x9 Gomoku and 19x19 Go, deleting the prior still leaves a large residual response rises from 13.8% to 26.3% in Gomoku and from 0.6% to 33.8% in Go-and soft reweighting teaches as well as hard action restriction, so pruning legal actions is not the mechanism. The same cross bounds the claim: re-enabling the prior restores nearly 100% response, leaving dependence gaps of 73.7 and 65.8 points. A latched-position evaluation localizes the residual to trained geometry-absent at the shared initialization, emerging over training, worth +17.3 points on in-distribution defenses but only +1.3 on structurally novel ones. Search is therefore best read as a training-time behavioral curriculum whose lessons are real, partial, and geometry-bound, and online competence and internalized competence are different estimands that a diagonal ablation cannot tell apart.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。