用机器学习压缩键值映射,大幅降低存储空间。
Learned Static Function Data Structures
- 用模型预测键对应的值分布,生成专属编码
- 实测在真实数据上节省90%以上空间
- 适合需要高效存储的键值查询场景
我们研究如何为静态键集构建键值映射的数据结构,支持未收录键的任意输出。与哈希表相比,这类静态函数数据结构无需存储键集,显著减少内存占用。已有方法中,压缩静态函数可逼近值序列的零阶经验熵。本文提出学习型静态函数,利用机器学习捕捉键与值间的相关性。对每个键,模型预测值的概率分布,据此生成键特定的前缀码,以紧凑方式编码真实值,再存入经典静态函数数据结构。该设计使学习型静态函数突破零阶熵限制,同时支持点查询。实验表明,真实数据下空间节省达一个数量级,合成数据下可达三个数量级。
原文摘要 · Abstract (English)
We consider the task of constructing a data structure for associating a static set of keys with values, while allowing arbitrary output values for queries involving keys outside the set. Compared to hash tables, these so-called static function data structures do not need to store the key set and thus use significantly less memory. Several techniques are known, with compressed static functions approaching the zero-order empirical entropy of the value sequence. In this paper, we introduce learned static functions, which use machine learning to capture correlations between keys and values. For each key, a model predicts a probability distribution over the values, from which we derive a key-specific prefix code to compactly encode the true value. The resulting codeword is stored in a classic static function data structure. This design allows learned static functions to break the zero-order entropy barrier while still supporting point queries. Our experiments show substantial space savings: up to one order of magnitude on real data, and up to three orders of magnitude on synthetic data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。