从基础原理推导出无需解码器的稀疏自编码器，实现可解释成分学习。
Deriving Decoder-Free Sparse Autoencoders from First Principles
- 基于LSE目标函数的梯度等于各成分责任，隐含期望-最大化机制。
- 无体积控制时模型会坍塌，方差和去相关性防止组件失效与冗余。
- 无需解码器，直接学习可解释的混合成分，适合特征可解释性研究。
在对数和指数（LSE）目标函数上进行梯度下降，会隐式执行期望-最大化（EM）：每个组件输出的梯度等于其责任值。该理论预测，若缺乏类似高斯混合模型中对数行列式的体积控制，将导致模型坍塌。我们在一个单层编码器中实现该理论，采用LSE目标函数并辅以InfoMax正则化实现体积控制。实验验证了理论预测：梯度-责任恒等关系精确成立；仅用LSE会导致坍塌；方差抑制死组件；去相关性防止冗余。模型表现出类EM优化动态，更低损失并不对应更优特征，自适应优化器无优势。最终得到的无解码器模型学习到可解释的混合成分，证实隐式EM理论可指导架构设计。
原文摘要 · Abstract (English)
Gradient descent on log-sum-exp (LSE) objectives performs implicit expectation--maximization (EM): the gradient with respect to each component output equals its responsibility. The same theory predicts collapse without volume control analogous to the log-determinant in Gaussian mixture models. We instantiate the theory in a single-layer encoder with an LSE objective and InfoMax regularization for volume control. Experiments confirm the theory's predictions. The gradient--responsibility identity holds exactly; LSE alone collapses; variance prevents dead components; decorrelation prevents redundancy. The model exhibits EM-like optimization dynamics in which lower loss does not correspond to better features and adaptive optimizers offer no advantage. The resulting decoder-free model learns interpretable mixture components, confirming that implicit EM theory can prescribe architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。