让深度模型通过压缩数据实现自我理解,而非仅预测。
A Theory of Machine Understanding via the Minimum Description Length Principle
- 用最小描述长度原则构建可精确评估的谱变分自编码器
- 发现最少的潜在激活模式能实现最优压缩
- 理论证明压缩到位时自然产生可解释表征,适合做可解释AI研究
过去十年,端到端训练的深度神经网络在多个领域取得显著成功。然而,这种黑箱式学习策略面临两大根本局限:难以自监督地形成可解释表征,且无法严格遵循最小描述长度(MDL)原则进行信息压缩。这两个问题指向更深层缺陷:端到端模型无法真正‘理解’所学内容。本文建立新理论,连接这两项局限。设计了谱变分自编码器(Spectrum VAE),其最小描述长度可严格计算。引入潜在维度组合(称作‘脉冲模式’),并证明在给定数据下,脉冲模式越少,越能实现MDL。理论表明,当模型在给定数据分布下达到最小描述长度时,将自然生成可解释的潜在表征。换言之,可解释表征——即‘理解’——可通过强制模型遵守MDL原则,在自监督下自发涌现。我们提出核心观点:理解即压缩。本理论主张深度网络训练目标应从仅最小化预测误差,扩展为同时最小化对数据的描述长度,使模型不仅学会,更能理解所学。本工作纯属理论,旨在启发未来基于MDL原则的自监督、可解释人工智能研究。
原文摘要 · Abstract (English)
Deep neural networks trained through end-to-end learning have achieved remarkable success across various domains in the past decade. However, the end-to-end learning strategy, originally designed to minimize predictive loss in a black-box manner, faces two fundamental limitations: the struggle to form explainable representations in a self-supervised manner, and the inability to compress information rigorously following the Minimum Description Length (MDL) principle. These two limitations point to a deeper issue: an end-to-end learning model is not able to "understand" what it learns. In this paper, we establish a novel theory connecting these two limitations. We design the Spectrum VAE, a novel deep learning architecture whose minimum description length (MDL) can be rigorously evaluated. Then, we introduce the concept of latent dimension combinations, or what we term spiking patterns, and demonstrate that the observed spiking patterns should be as few as possible based on the training data in order for the Spectrum VAE to achieve the MDL. Finally, our theory demonstrates that when the MDL is achieved with respect to the given data distribution, the Spectrum VAE will naturally produce explainable latent representations of the data. In other words, explainable representations--or "understanding"--can emerge in a self-supervised manner simply by making the deep network obey the MDL principle. In our opinion, this also implies a deeper insight: To understand is to compress. At its core, our theory advocates for a shift in the training objective of deep networks: not only to minimize predictive loss, but also to minimize the description length regarding the given data. That is, a deep network should not only learn, but also understand what it learns. This work is entirely theoretical and aims to inspire future research toward self-supervised, explainable AI grounded in the MDL principle.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。