将Dropout转化为可调控的确定性正则项，提升Transformer模型稳定性。
Explicit Dropout: Deterministic Regularization for Transformer Architectures
- 把随机掩码转为显式正则项，直接加到损失函数中
- 在图像、音频、动作检测任务中表现优于传统Dropout
- 可独立调节注意力与前馈层的正则强度，适合需要可控性的场景
Dropout是深度学习中广泛使用的正则化技术，但其效果通常依赖随机掩码实现。本文提出一种确定性公式，将Dropout表示为可直接融入训练损失的加性正则项。该框架为Transformer架构中的注意力查询、键、值及前馈组件推导出显式正则项，各部分正则强度可独立控制。此方法摆脱对随机扰动的依赖，提供更清晰、细粒度的正则化控制。在图像分类、时序动作检测和音频分类任务上的实验表明，显式Dropout在性能上匹配或超越传统隐式方法，且在注意力层和前馈网络中均带来一致增益。消融实验显示，通过正则系数和丢弃率可稳定控制正则强度。整体而言，显式Dropout提供了实用且可解释的替代方案，在多种任务中保持架构灵活性。
原文摘要 · Abstract (English)
Dropout is a widely used regularization technique in deep learning, but its effects are typically realized through stochastic masking rather than explicit optimization objectives. We propose a deterministic formulation that expresses dropout as an additive regularizer directly incorporated into the training loss. The framework derives explicit regularization terms for Transformer architectures, covering attention query, key, value, and feed-forward components with independently controllable strengths. This formulation removes reliance on stochastic perturbations while providing clearer and fine-grained control over regularization strength. Experiments across image classification, temporal action detection, and audio classification show that explicit dropout matches or outperforms conventional implicit methods, with consistent gains when applied to attention and feed-forward network layers. Ablation studies demonstrate stable performance and controllable regularization through regularization coefficients and dropout rates. Overall, explicit dropout offers a practical and interpretable alternative to stochastic regularization while maintaining architectural flexibility across diverse tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。