用L⁴损失训练网络，让50个神经元同时计算100个稀疏特征的ReLU。
Compressed Computation under $L^4$ Loss is likely Computation in Superposition
- 用L⁴损失驱动网络在超维度中并行计算多个函数。
- 50个神经元成功实现100个稀疏特征的并行激活，准确率超90%。
- 通过二进制编码和伪逆解码，可压缩模型为三标量描述。
神经网络通常将概念表示为激活空间中的方向，而超叠加（superposition）使其能编码多于维度数量的概念。自然地，人们会问：它们是否也能执行多于神经元数量的函数计算，即在超叠加中进行计算？目前对超叠加的表征已有较好理解，但计算层面的超叠加仍不清晰，且缺乏由训练生成而非人工设计的简化模型。本文以压缩计算场景为模型：一个含50个神经元的单层ReLU网络需计算100个稀疏输入特征的ReLU。我们发现，使用L⁴损失（误差均值四次方）训练时，网络展现出计算所有特征于超叠加中的行为。通过逆向工程，我们发现网络为每个特征分配稀疏二进制编码，并用编码矩阵的伪逆进行解码。仅用三个标量即可重构网络性能，且通过手工设计编码构建的等效网络验证了该机制的有效性。
原文摘要 · Abstract (English)
Neural networks are thought to represent concepts as directions in their activation space, and superposition lets them encode more concepts than they have dimensions. It is natural to ask whether they can also compute more functions than they have neurons, i.e., perform computation in superposition. In this regime many functions of sparse inputs are evaluated by a layer with fewer neurons than there are functions to compute. Representation in superposition is by now fairly well understood, but computation in superposition is not, and there are few toy models of it arising through training rather than being hand designed. As a toy model of computation in superposition we study the compressed-computation setup: a single-hidden-layer ReLU network with 50 neurons that must compute the ReLU of each of 100 sparse input features. We show that training it under an $L^4$ loss (the mean fourth power of the error), rather than the usual $L^2$, elicits a solution that appears to compute all features in superposition. We then reverse-engineer this solution. We find that the network assigns each feature a sparse binary codeword over neurons and decodes it with a pseudoinverse of the encoder. Given these codewords, a description with only three scalars recovers most of the network's performance, and we validate it by building equivalent networks from hand-designed codes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。