根据特征维度确定软最大温度,无需训练即可提升分类性能。
Analytical Softmax Temperature Setting from Feature Dimensions for Model- and Domain-Robust Classification
- 从特征维度推导出最优温度理论公式,实现无训练设置。
- 提出校正方案,使温度适应类别数与任务复杂度,提升泛化能力。
- 插入批归一化稳定特征空间,适合跨模型跨数据集应用。
在深度学习分类任务中,软最大函数的温度参数 $T$ 对输出分布和整体性能有关键影响。本文提出新理论:最优温度 $T^*$ 由特征表示的维度唯一决定,从而实现无需训练即可确定 $T^*$。尽管理论基础坚实,但实际中 $T^*$ 受模型、数据集等干扰因素影响而波动。为此,我们设计并优化一组温度调整系数,依据特征维度理论关系动态调节 $T^*$。同时,在输出层前插入批归一化层,有效稳定特征空间。结合大规模实验,我们提出一个经验公式,在不需额外训练的前提下估计 $T^*$,并引入修正机制以根据类别数和任务复杂度进一步优化 $T^*$。实验验证所推导温度不仅符合理论预期,且在多种任务中表现一致提升,为 $T^*$ 提供一种实用、无需训练的解决方案。
原文摘要 · Abstract (English)
In deep learning-based classification tasks, the softmax function's temperature parameter $T$ critically influences the output distribution and overall performance. This study presents a novel theoretical insight that the optimal temperature $T^*$ is uniquely determined by the dimensionality of the feature representations, thereby enabling training-free determination of $T^*$. Despite this theoretical grounding, empirical evidence reveals that $T^*$ fluctuates under practical conditions owing to variations in models, datasets, and other confounding factors. To address these influences, we propose and optimize a set of temperature determination coefficients that specify how $T^*$ should be adjusted based on the theoretical relationship to feature dimensionality. Additionally, we insert a batch normalization layer immediately before the output layer, effectively stabilizing the feature space. Building on these coefficients and a suite of large-scale experiments, we develop an empirical formula to estimate $T^*$ without additional training while also introducing a corrective scheme to refine $T^*$ based on the number of classes and task complexity. Our findings confirm that the derived temperature not only aligns with the proposed theoretical perspective but also generalizes effectively across diverse tasks, consistently enhancing classification performance and offering a practical, training-free solution for determining $T^*$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。