揭秘神经网络输出与损失函数的统计关联，教你选对损失函数。
DL101 Neural Network Outputs and Loss Functions
- 从最大似然原理出发，解释不同输出层激活函数的数学依据。
- 损失函数选择等价于假设输出服从特定概率分布，如MSE对应正态分布。
- 适合深度学习初学者和需要严谨建模的科研人员阅读。
从统计学角度分析神经网络输出层常用激活函数（如线性、Sigmoid、ReLU、Softmax）的数学性质及其适用场景。报告阐明，常见的损失函数（如均方误差MSE、平均绝对误差MAE、各类交叉熵）均可通过最大似然估计（MLE）理论推导得出，其选择等价于对模型输出假设特定的概率分布，揭示了损失函数与广义线性模型（GLMs）之间的深层联系。此外还讨论了实际应用中可能遇到的输出编码方式、输出约束及重尾分布等情形。
原文摘要 · Abstract (English)
The loss function used to train a neural network is strongly connected to its output layer from a statistical point of view. This technical report analyzes common activation functions for a neural network output layer, like linear, sigmoid, ReLU, and softmax, detailing their mathematical properties and their appropriate use cases. A strong statistical justification exists for the selection of the suitable loss function for training a deep learning model. This report connects common loss functions such as Mean Squared Error (MSE), Mean Absolute Error (MAE), and various Cross-Entropy losses to the statistical principle of Maximum Likelihood Estimation (MLE). Choosing a specific loss function is equivalent to assuming a specific probability distribution for the model output, highlighting the link between these functions and the Generalized Linear Models (GLMs) that underlie network output layers. Additional scenarios of practical interest are also considered, such as alternative output encodings, constrained outputs, and distributions with heavy tails.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。