用字节编码子词，隐私保护不降性能
Subword Embedding from Bytes Gains Privacy without Sacrificing Accuracy and Complexity
- 将子词转为字节序列，阻断文本还原攻击
- 仅需256字节词汇表，内存占用更少
- 适合对隐私敏感的联邦学习场景
尽管自然语言处理模型深刻影响日常生活，但隐私泄露问题日益突出。尽管联邦学习提升了隐私保护，攻击者仍可能通过模型参数和梯度恢复训练数据。为此，我们提出子词到字节的嵌入方法（SEB），利用深度神经网络将子词编码为字节序列，显著增加原始文本恢复难度。该方法仅需256字节的词汇表，同时保持与标准方法相同的输入长度和效率。实验表明，SEB在联邦学习中有效抵御基于嵌入的攻击，且在机器翻译、情感分析和语言建模任务中表现媲美甚至优于传统子词嵌入方法，同时具备更低的时间和空间复杂度。
原文摘要 · Abstract (English)
While NLP models significantly impact our lives, there are rising concerns about privacy invasion. Although federated learning enhances privacy, attackers may recover private training data by exploiting model parameters and gradients. Therefore, protecting against such embedding attacks remains an open challenge. To address this, we propose Subword Embedding from Bytes (SEB) and encode subwords to byte sequences using deep neural networks, making input text recovery harder. Importantly, our method requires a smaller memory with $256$ bytes of vocabulary while keeping efficiency with the same input length. Thus, our solution outperforms conventional approaches by preserving privacy without sacrificing efficiency or accuracy. Our experiments show SEB can effectively protect against embedding-based attacks from recovering original sentences in federated learning. Meanwhile, we verify that SEB obtains comparable and even better results over standard subword embedding methods in machine translation, sentiment analysis, and language modeling with even lower time and space complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。