CC模型看似超叠加计算，实则依赖输入混杂机制。
Compressed Computation is (probably) not Computation in Superposition
- 通过分离损失项发现，性能提升源于输入混杂而非真正并行计算。
- 模型学习到的神经元方向集中于混杂矩阵的前50个特征值子空间。
- 仅用混杂矩阵构造的基线模型即可复现主要现象，适合研究混淆效应。
我们研究压缩计算（CC）模型是否属于超叠加计算。该模型看似用50个神经元计算100个ReLU函数，且损失优于仅表示50个函数的预期。分析发现，模型通过噪声残差流混合输入，导致标签中出现意外的混杂矩阵。将训练目标拆分为ReLU项与混杂项后，性能提升与混杂矩阵幅度正相关，移除该矩阵后性能消失。学习到的神经元方向集中于混杂矩阵的前50个最大特征值对应子空间，表明混杂项主导解结构。进一步构建仅基于混杂矩阵的半非负矩阵分解（SNMF）基线，可复现相似的损失曲线，并优于先前基线，但未达到训练模型水平。结果表明，CC并非超叠加计算的合适模型。
原文摘要 · Abstract (English)
We study whether the Compressed Computation (CC) toy model (Braun et al., 2025) is an instance of computation in superposition. The CC model appears to compute 100 ReLU functions with just 50 neurons, achieving a better loss than expected from only representing 50 ReLU functions. We show that the model mixes inputs via its noisy residual stream, corresponding to an unintended mixing matrix in the labels. Splitting the training objective into the ReLU term and the mixing term, we find that performance gains scale with the magnitude of the mixing matrix and vanish when the matrix is removed. The learned neuron directions concentrate in the subspace associated with the top 50 eigenvalues of the mixing matrix, suggesting that the mixing term governs the solution. Finally, a semi-non-negative matrix factorization (SNMF) baseline derived solely from the mixing matrix reproduces the qualitative loss profile and improves on prior baselines, though it does not match the trained model. These results suggest CC is not a suitable toy model of computation in superposition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。