让强化学习策略通过可解释概念实现零样本迁移，提升跨算法协作效率。
Prism: Policy Reuse via Interpretable Strategy Mapping in Reinforcement Learning
- 用聚类提取策略中的离散概念，并通过因果干预验证其驱动作用。
- 概念迁移使不同算法训练的智能体在围棋上胜率超76%，远超随机水平。
- 适用于策略状态天然离散的领域，如围棋，不适用于连续动作场景。
我们提出PRISM（Policy Reuse via Interpretable Strategy Mapping），一个将强化学习智能体决策基于离散、因果验证概念的框架，并利用这些概念作为不同算法训练智能体间的零样本迁移接口。PRISM通过K-means将每个智能体的编码器特征聚类为K个概念。因果干预表明这些概念直接驱动行为：在2500次干预中，改变概念分配导致动作变化的比例达69.4%（p = 8.6 × 10^-86）。概念重要性与使用频率解耦：最常用的概念C47（33.0%频率）被移除时仅导致胜率下降9.4%，而移除使用率15.4%的C16则使胜率从100%降至51.8%。由于概念因果编码了策略，通过最优二分图匹配对齐概念，实现零样本策略知识迁移。在Go 7×7上，三个独立训练的智能体间概念迁移在两组成功配对中分别达到69.5%±3.2%和76.4%±3.4%胜率（10种子），远高于随机智能体的3.5%和未对齐情况下的9.2%。迁移成功依赖源策略强度；几何对齐质量无法预测性能（R² ≈ 0）。该框架限定于策略状态天然离散的领域：在Atari Breakout上，相同流程产生瓶颈策略，表现等同于随机智能体，证实了围棋结果反映的是领域结构特性。
原文摘要 · Abstract (English)
We present PRISM (Policy Reuse via Interpretable Strategy Mapping), a framework that grounds reinforcement learning agents' decisions in discrete, causally validated concepts and uses those concepts as a zero-shot transfer interface between agents trained with different algorithms. PRISM clusters each agent's encoder features into $K$ concepts via K-means. Causal intervention establishes that these concepts directly drive - not merely correlate with - agent behavior: overriding concept assignments changes the selected action in 69.4% of interventions ($p = 8.6 \times 10^{-86}$, 2500 interventions). Concept importance and usage frequency are dissociated: the most-used concept (C47, 33.0% frequency) causes only a 9.4% win-rate drop when ablated, while ablating C16 (15.4% frequency) collapses win rate from 100% to 51.8%. Because concepts causally encode strategy, aligning them via optimal bipartite matching transfers strategic knowledge zero-shot. On Go~7$\times$7 with three independently trained agents, concept transfer achieves 69.5%$\pm$3.2% and 76.4%$\pm$3.4% win rate against a standard engine across the two successful transfer pairs (10 seeds), compared to 3.5% for a random agent and 9.2% without alignment. Transfer succeeds when the source policy is strong; geometric alignment quality predicts nothing ($R^2 \approx 0$). The framework is scoped to domains where strategic state is naturally discrete: the identical pipeline on Atari Breakout yields bottleneck policies at random-agent performance, confirming that the Go results reflect a structural property of the domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。