通过伯努利丢弃研究编码器-解码器表示，发现性能有临界阈值。
Understanding Transformer Encoder-Decoder Representations through Bernoulli Dropout
- 在编码器与解码器间施加伯努利丢弃，调节保留概率
- 验证准确率和BLEU分数在某阈值处骤降，存在性能临界点
- 揭示高维嵌入中稀疏性对模型鲁棒性的关键影响
我们从高维编码器-解码器嵌入的角相似性角度研究Transformer的过参数化问题。通过在编码器与解码器之间施加伯努利丢弃，调整保留概率 $p$，识别出保持Top-1预测性能的稀疏性阈值。理论上证明：若有效稀疏嵌入足够大，则解码器性能在中等坐标丢弃下仍保持稳定。实验上，构建带有二元擦除信道（BEC）的新型Transformer模型，在英法翻译任务上测试，结果可视化显示验证准确率与BLEU分数均在某阈值处急剧下降。
原文摘要 · Abstract (English)
We study Transformer overparameterization through the lens of angular similarity in high-dimensional encoder-decoder embeddings. We apply Bernoulli dropout between the encoder and the decoder, varying the keep probability $p$ to identify a sparsity-dependent threshold above which the Top-1 prediction is preserved. Theoretically, we prove that, if the effective sparsity embeddings is sufficiently large, and thus decoder performance, remain stable under moderate coordinate dropout. Empirically, we implement the Bernoulli dropout by constructing a new Transformer model augmented with Binary Erasure Channel (BEC) and test its performance on an English-French translation task. Experimental results visualize the trends for validation accuracies and BLEU scores, both decline sharply at some threshold.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。