用信息论分析神经网络如何逐层传递信息，揭示优化策略本质。
Information flow in multilayer perceptrons: an in-depth analysis
- 引入信息矩阵框架，刻画各层信息流动机制
- 发现信息瓶颈优化策略与该框架高度一致
- 模型本质是按目标适配输入的
分析多层感知机中信息沿层流动的机制，是人工神经网络领域的重要课题。本文从信息论视角出发，针对监督学习的约束条件，深入研究信息处理方式。为此提出信息矩阵概念，作为理解优化策略成因和信息流动的理论框架。研究取得三项关键成果：一、定义了一种参数化优化策略；二、发现信息瓶颈框架中的优化策略与信息矩阵推导结果高度相似；三、揭示多层感知机本质上是一种“适配器”，根据给定目标对输入进行处理。
原文摘要 · Abstract (English)
Analysing how information flows along the layers of a multilayer perceptron is a topic of paramount importance in the field of artificial neural networks. After framing the problem from the point of view of information theory, in this position article a specific investigation is conducted on the way information is processed, with particular reference to the requirements imposed by supervised learning. To this end, the concept of information matrix is devised and then used as formal framework for understanding the aetiology of optimisation strategies and for studying the information flow. The underlying research for this article has also produced several key outcomes: i) the definition of a parametric optimisation strategy, ii) the finding that the optimisation strategy proposed in the information bottleneck framework shares strong similarities with the one derived from the information matrix, and iii) the insight that a multilayer perceptron serves as a kind of "adaptor", meant to process the input according to the given objective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。