通过约束梯度方向提升模型置信度可靠性，无需修改网络结构。
OrthoGrad Improves Neural Calibration
- 在梯度更新中强制与权重向量正交，改变优化路径。
- 在CIFAR-10仅10%数据下，准确率相当但测试损失和熵值显著降低。
- 适合作为高不确定性场景下的通用优化器改进方案。
我们研究了$\perp$Grad，一种几何感知的梯度优化改进方法，通过约束下降方向来缓解标准优化器在关键不确定性场景中的过度自信问题。该方法通过强制梯度更新与权重向量正交，在不改变网络结构的前提下调整优化轨迹。在使用10%标注数据的CIFAR-10上，$\perp$Grad在准确率上与SGD相当，同时在测试损失（p=0.05）、预测熵（p=0.001）和置信度指标上均实现统计显著改善。这些效果在不同数据损坏程度和网络架构间保持一致。$\perp$Grad对优化器无依赖，开销极低，且兼容事后校准技术。理论上，我们分析了简化版$\perp$Grad的收敛性和稳定点，揭示正交化能限制损失下降路径，防止置信度虚高，促进决策边界优化。结果表明，低计算成本的几何干预可有效提升预测不确定性估计质量。
原文摘要 · Abstract (English)
We study $\perp$Grad, a geometry-aware modification to gradient-based optimization that constrains descent directions to address overconfidence, a key limitation of standard optimizers in uncertainty-critical applications. By enforcing orthogonality between gradient updates and weight vectors, $\perp$Grad alters optimization trajectories without architectural changes. On CIFAR-10 with 10% labeled data, $\perp$Grad matches SGD in accuracy while achieving statistically significant improvements in test loss ($p=0.05$), predictive entropy ($p=0.001$), and confidence measures. These effects show consistent trends across corruption levels and architectures. $\perp$Grad is optimizer-agnostic, incurs minimal overhead, and remains compatible with post-hoc calibration techniques. Theoretically, we characterize convergence and stationary points for a simplified $\perp$Grad variant, revealing that orthogonalization constrains loss reduction pathways to avoid confidence inflation and encourage decision-boundary improvements. Our findings suggest that geometric interventions in optimization can improve predictive uncertainty estimates at low computational cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。