用轻量模型实现高速文本嵌入,兼顾精度与效率。
Luxical: High-Speed Lexical-Dense Text Embeddings

- 结合TF-IDF与小规模神经网络,通过知识蒸馏逼近大模型效果。
- 在网页爬取和数据清洗任务中实现3到100倍加速,性能接近神经网络基线。
- 适合需要大规模文本组织的场景,如数据清洗与检索系统。
前沿语言模型的质量越来越依赖于对海量文本语料的组织能力。当前主流工具在速度与灵活性间权衡:词汇分类器(如FastText)速度快但仅输出分类分数,而基于Transformer的文本嵌入模型虽灵活支持聚类、分类、检索等多种工作流,但计算成本高。我们提出Luxical,一个用于高速“词法-稠密”文本嵌入的开源库,旨在融合两者优势。Luxical结合稀疏TF-IDF特征、小型ReLU网络与知识蒸馏训练策略,在远低于神经网络的开销下逼近大型Transformer嵌入模型的效果。本技术报告描述了Luxical架构与训练目标,并在两项不同任务中评估其性能:针对性网页爬虫文档检索测试与基于文本分类的语言模型数据清洗任务。实验显示,相比不同规模的神经基线,提速达3至100倍;在数据清洗任务中,推理速度与FastText相当。所测模型在大规模文本组织中展现出良好的计算/质量权衡,性能可媲美神经基线。Luxical已开源,地址为https://github.com/datologyai/luxical。
原文摘要 · Abstract (English)
Frontier language model quality increasingly hinges on our ability to organize web-scale text corpora for training. Today's dominant tools trade off speed and flexibility: lexical classifiers (e.g., FastText) are fast but limited to producing classification output scores, while the vector-valued outputs of transformer text embedding models flexibly support numerous workflows (e.g., clustering, classification, and retrieval) but are computationally expensive to produce. We introduce Luxical, a library for high-speed "lexical-dense" text embeddings that aims to recover the best properties of both approaches for web-scale text organization. Luxical combines sparse TF--IDF features, a small ReLU network, and a knowledge distillation training regimen to approximate large transformer embedding models at a fraction of their operational cost. In this technical report, we describe the Luxical architecture and training objective and evaluate a concrete Luxical model in two disparate applications: a targeted webcrawl document retrieval test and an end-to-end language model data curation task grounded in text classification. In these tasks we demonstrate speedups ranging from 3x to 100x over varying-sized neural baselines, and comparable to FastText model inference during the data curation task. On these evaluations, the tested Luxical model illustrates favorable compute/quality trade-offs for large-scale text organization, matching the quality of neural baselines. Luxical is available as open-source software at https://github.com/datologyai/luxical.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。