Tiny Monolingual Encoders 用高效训练方法打造轻量语言模型，兼顾性能与能耗。
TiME: Tiny Monolingual Encoders for Efficient NLP Pipelines
- 采用知识蒸馏技术，从大模型压缩出小而快的单语编码器。
- 在多项任务中实现更高吞吐、更低延迟和更少能耗，性能不降。
- 支持低资源语言，适合移动端或实时应用部署。
当前大量语言模型研究聚焦于大型通用模型，但许多NLP流水线仅需具备特定功能的小型模型。尽管大模型能完成这些任务，却因速度慢、能耗高，难以处理大规模数据或实现实时响应，且不利于可持续发展及在电池设备上的部署。本文提出TiME（Tiny Monolingual Encoders），通过现代训练技术如知识蒸馏，构建高效小型模型。与众多现成NLP流水线不同，我们的模型支持低资源语言，并在多个常见NLP任务上进行全面评估，展现出在基准性能、吞吐量、延迟和能耗之间更优的权衡。同时证明：可将单语教师模型蒸馏为单语学生模型；也可将使用绝对位置编码的模型从采用相对位置编码的教师模型中蒸馏出来。
原文摘要 · Abstract (English)
Today, a lot of research on language models is focused on large, general-purpose models. However, many NLP pipelines only require models with a well-defined, small set of capabilities. While large models are capable of performing the tasks of those smaller models, they are simply not fast enough to process large amounts of data or offer real-time responses. Furthermore, they often use unnecessarily large amounts of energy, leading to sustainability concerns and problems when deploying them on battery-powered devices. In our work, we show how to train small models for such efficiency-critical applications. As opposed to many off-the-shelf NLP pipelines, our models use modern training techniques such as distillation, and offer support for low-resource languages. We call our models TiME (Tiny Monolingual Encoders) and comprehensively evaluate them on a range of common NLP tasks, observing an improved trade-off between benchmark performance on one hand, and throughput, latency and energy consumption on the other. Along the way, we show that distilling monolingual models from multilingual teachers is possible, and likewise distilling models with absolute positional embeddings from teachers with relative positional embeddings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。