提出SDM激活函数，提升模型决策可解释性与鲁棒性。
Similarity-Distance-Magnitude Activations
- 引入相似性、距离、幅度三重感知的新型激活函数
- 在分布外数据下准确率优于传统softmax校准方法
- 适合需要可信预测与可解释性的下游应用
我们提出相似性-距离-幅度（SDM）激活函数，相较于标准softmax，增加了对训练中正确深度匹配的相似性感知、对训练分布的距离感知，以及对决策边界幅度的感知，通过密集匹配实现基于实例的可解释性。进一步提出基于数据驱动的类别经验累积分布函数分区的SDM估计器，用于控制选择性分类中的类别和预测条件下的准确率。当作为预训练语言模型的最终层激活函数用于选择性分类时，该方法在协变量偏移和分布外输入下比使用softmax激活的传统校准方法更具鲁棒性，同时保持对分布内数据的信息量。
原文摘要 · Abstract (English)
We introduce the Similarity-Distance-Magnitude (SDM) activation function, a more robust and interpretable formulation of the standard softmax activation function, adding Similarity (i.e., correctly predicted depth-matches into training) awareness and Distance-to-training-distribution awareness to the existing output Magnitude (i.e., decision-boundary) awareness, and enabling interpretability-by-exemplar via dense matching. We further introduce the SDM estimator, based on a data-driven partitioning of the class-wise empirical CDFs via the SDM activation, to control the class- and prediction-conditional accuracy among selective classifications. When used as the final-layer activation over pre-trained language models for selective classification, the SDM estimator is more robust to covariate shifts and out-of-distribution inputs than existing calibration methods using softmax activations, while remaining informative over in-distribution data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。