随机初始化的CNN在强化学习中自发产生极稀疏表示，揭示任务内在结构。
Emergent Sparsity in Frozen Random CNN Feature Extractors for Deep Reinforcement Learning
- 用冻结的随机CNN提取特征，无需显式稀疏目标即出现神经元高度稀疏。
- 不同任务激活神经元数从1-3（Pong）到42（太空侵略者），反映任务复杂度。
- 稀疏性由输入冗余决定，适合研究模型压缩与任务本质结构的学者。
我们发现：使用冻结的随机初始化卷积神经网络（CNN）作为特征提取器的深度强化学习代理，会自发产生极稀疏的全连接层表示，且不依赖任何稀疏性诱导目标。在第一层全连接层（FC1，3,136→64）中，确定性Pong仅需1-3个神经元激活，随机性Pong为5-11个，而可训练的CNN则激活55-64个神经元。我们得出四项核心结论：第一，FC1稀疏性随任务复杂度上升：Pong为1-11，Breakout为19-26，太空侵略者约42；宽度缩放验证其反映任务结构而非固定容量比例。第二，同一游戏内稀疏性存在差异：三个相同种子的Pong分别激活5、7、11个神经元，5神经元版本止步于+14奖励，其余达专家水平（+18.4，+18.7），表明随机投影的可用维度限制性能上限。第三，消融实验显示移除这些活跃神经元会导致两个PPO实现和四种游戏性能崩溃。第四，信息瓶颈早期形成：主动神经元集合在15-30M步锁定，而奖励转正发生在35-105M步后。在Breakout中，冻结和可训练的CNN以不同结构达到相近奖励：冻结代理使用17-25个活跃神经元（参与比~10-14），可训练代理使用51个（参与比~3.6）。最终，在输入维度远超任务内在维度时，对冻结随机投影的梯度下降可揭示问题的有效秩，无需显式稀疏机制。
原文摘要 · Abstract (English)
We report a striking phenomenon: deep reinforcement learning agents trained with frozen, randomly initialized CNN feature extractors spontaneously develop extremely sparse fully-connected representations, without any sparsity-inducing objective. In the first fully-connected layer (FC1, $3{,}136 \to 64$), agents compress task-relevant information through as few as 1-3 neurons out of 64 for deterministic Pong (5-11 for stochastic Pong), while trainable CNNs activate 55-64 neurons under matched conditions. We establish four principal findings. First, FC1 sparsity scales with task complexity: 1-11 for Pong, 19-26 for Breakout, and $\sim$42 for Space Invaders. Width-scaling confirms this reflects task structure rather than a fixed capacity fraction. Second, within-game scaling emerges: three identical Pong seeds produce 5, 7, and 11 active neurons. The 5-neuron seed plateaus at $+14$ reward, while the others reach expert performance ($+18.4$, $+18.7$), suggesting the random projection's usable dimensionality bounds achievable performance. Third, ablation confirms necessity: removing these active neurons crashes performance across two PPO implementations and four games. Fourth, the information bottleneck commits early: a sweep shows the active set locks by 15-30M steps, while reward turns positive 35-105M steps later. A complementary finding in Breakout shows frozen and trainable CNNs reach competitive rewards via structurally different bottlenecks: frozen agents use 17-25 active neurons (participation ratio $\sim$10-14), while trainable agents use 51 (participation ratio $\sim$3.6). Finally, wherever input dimensionality dwarfs intrinsic task dimensionality, gradient descent on a frozen random projection may reveal the effective rank of the underlying problem without explicit sparsity machinery.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。