arXiv:2506.01254cs.CL2025-06

用双数组字典树压缩FastText,大幅降内存还提速

Memory-Efficient FastText: A Comprehensive Approach Using Double-Array Trie Structures and Mark-Compact Memory Management

  • 用双数组字典树替代哈希表,避免无意义的词干碰撞
  • 在3010万词、2.87亿词干上,内存从145.2GB降至28.9GB
  • 适合需要高效词向量存储的大型语言模型检索系统

FastText因其能通过字符n-gram合成未登录词向量,仍是工业级词表示的实用选择。但其原始哈希桶实现存在两个工程妥协:无关n-gram会碰撞至同一行,增加桶数又使输入矩阵成为主要内存开销。本文提出一种基于‘精确再压缩’原则的内存优化FastText变体:先为每个观测到的词和n-gram分配唯一标识,再仅对学习向量与词法结构相似的行进行压缩。具体地,以无冲突的双数组字典树索引替代哈希桶,并通过结构约束的前缀/后缀合并及标记紧凑化行重组压缩结果矩阵。相比任意哈希,该方法仅在余弦相似度高时才共享行,保留可解释的n-gram身份,同时减少活跃行数。我们描述了完整的训练与服务流程,包括支持UTF-8的n-gram枚举、双数组字典树查找、内存映射模型加载及词句向量重构。在一个含3010万词、2.874亿提取n-gram的中文大规模基准上,压缩模型将内存从145.2GB降至28.9GB,加载时间由12.3分钟缩短至3.2分钟,下游性能仅比无哈希模型低0.3分。我们将其定位为大语言模型时代的紧凑词汇记忆层,并开源为扩展版FastText原型。

原文摘要 · Abstract (English)

FastText remains a practical choice for industrial word representation because it can synthesize vectors for out-of-vocabulary words from character n-grams. Its original hash-bucket implementation, however, couples two engineering compromises that become painful at large scale: unrelated n-grams collide into the same row, while increasing the bucket count quickly turns the input matrix into the dominant memory cost. This paper presents a memory-efficient FastText variant based on an exact-then-compress principle: first give every observed word and n-gram an explicit identity, then compress only those rows whose learned vectors and lexical structure justify sharing. Concretely, we replace hash buckets with collision-free double-array trie indexes and compress the resulting n-gram matrix through structurally constrained prefix and suffix merging followed by mark-compact row reorganization. Unlike arbitrary hashing, the proposed method shares rows only after a high cosine-similarity test, preserving interpretable n-gram identities while reducing the number of live rows. We describe the full training and serving pipeline, including UTF-8 aware n-gram enumeration, double-array trie lookup, memory-mapped model loading, and vector reconstruction for words and sentences. On a large Chinese vocabulary benchmark with 30.1M words and 287.4M extracted n-grams, the compressed model reduces memory from 145.2GB to 28.9GB, improves load time from 12.3 minutes to 3.2 minutes, and preserves downstream quality within 0.3 points of a hash-free model. We position the result as a compact lexical memory layer for LLM-era retrieval systems and release the implementation as an extended FastText prototype.

词向量内存优化字典树LLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。