用差分隐私训练小规模Transformer，提升输入预测准确率。
Privacy-Preserving Transformers: SwiftKey's Differential Privacy Implementation
- 缩小GPT2架构+两阶段训练：先通用数据预训练，再用差分隐私微调打字数据。
- 相比生产级GRU模型，下一词预测准确率小幅提升，内存与速度开销可控增长。
- 适用于注重隐私保护的移动端输入法场景，兼顾效率与安全性。
本文在SwiftKey中使用差分隐私（DP）训练Transformer进行语言建模。通过多组实验平衡模型大小、运行速度与准确性之间的权衡。结果显示，相较于生产级的GRU模型，该方法在保持可接受内存与速度开销的前提下，实现了下一词预测准确率的小幅且稳定提升。具体做法是将GPT2架构缩至合适规模，并采用两阶段训练流程：首先在通用数据上构建初始模型，再在打字数据上进行差分隐私微调。最终模型通过ONNX集成，兼具灵活性与高效性。
原文摘要 · Abstract (English)
In this paper we train a transformer using differential privacy (DP) for language modeling in SwiftKey. We run multiple experiments to balance the trade-off between the model size, run-time speed and accuracy. We show that we get small and consistent gains in the next-word-prediction and accuracy with graceful increase in memory and speed compared to the production GRU. This is obtained by scaling down a GPT2 architecture to fit the required size and a two stage training process that builds a seed model on general data and DP finetunes it on typing data. The transformer is integrated using ONNX offering both flexibility and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。