发现彩票票券本质是特征空间中的兼容代码家族,而非具体权重子网。
Toy Combinatorial Interpretability Models Reveal Lottery Tickets in Early Feature Space

- 在可解释的组合设置中,通过特征空间距离定位初始接近最终编码的位置。
- 胜出子网对应特征空间中靠近终态编码且干扰低的候选位置群。
- 适合研究模型内部机制或彩票票券本质的学者参考。
彩票票券假说认为密集网络中存在稀疏子网(‘胜出票券’),其在初始化权重回溯后独立训练,性能可媲美完整模型。我们提出更深层问题:胜出票券究竟保留了什么内部结构?在一种具有可解释特征表示的组合性、命题结构化小规模设定中,我们发现胜出票券在权重空间中的对应物,是特征空间中初始化时已接近最终特征通道编码的位置。密集随机梯度下降通过结构化选择实现这些位置的优化:邻近位置要么收敛至终态编码,要么被排斥,排斥集中在更拥挤的神经元上,暗示超叠加下的竞争机制。因此,胜出票券本质上是一组兼容的编码位置集合,共同平衡与终态编码的距离和低特征间干扰。稀疏重训常在同一类命题/模板家族中于不同行重新表达,表明被保留的是家族级而非微观行身份。我们基于特征空间距离与运动设计轻量探测器,在本设定下其准确率与编码恢复精度常优于现有基于权重的票券发现方法。尽管结果基于小规模设定,但提示彩票票券结构受隐藏特征空间几何支配,而非权重空间子网身份。
原文摘要 · Abstract (English)
The lottery ticket hypothesis posits that dense networks contain sparse subnetworks, ``winning tickets,'' that, when rewound to their initial weights and retrained in isolation, match the performance of the full model. We ask a more mechanistic question: what internal object does a winning ticket preserve? We work in a combinatorial, clause-structured toy setting that admits an interpretable feature-space representation with well-defined combinatorial distances between features. We show that winning tickets in weight space correspond to precursor locations in feature space that are already near, at initialization, to the final feature-channel codes. Dense SGD resolves these locations through structured selection: proximal locations either converge to final codes or are rejected, with rejection concentrated at more crowded neurons, implicating competition under superposition. A winning ticket is thus a family of compatible code locations that jointly balance proximity to final codes with low inter-feature interference. Sparse retraining often re-expresses the same clause/template family on a different row, so the preserved object is family-level rather than microscopic row identity. We validate this account with lightweight probes based on feature-space distance and motion; in our setting, these probes frequently outperform established weight-based ticket discovery methods in both accuracy and exact code recovery. Although these findings are grounded in a toy setting, they suggest that the lottery ticket structure is governed by hidden feature-space geometry rather than weight-space subnetwork identity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。