一种通用压缩搜索算法,可在不完全解压数据的前提下高效查找相似项。
Generalized compression and compressive search of large datasets
- 基于流形假设,仅解压相关部分实现压缩搜索
- 压缩率接近gzip,k-NN/ρ-NN搜索时间亚线性
- 适用于基因组、蛋白质组等大规模数据,开源可用
大数据爆炸催生了需在时间和内存上亚线性扩展的搜索算法。尽管压缩与搜索算法各自存在,但能同时实现二者的极少,且多局限于特定领域。本文提出panCAKES,一种新型压缩搜索方法,可在仅解压数据中少量相关部分的情况下,对压缩数据执行k-NN和ρ-NN搜索。panCAKES基于流形假设,利用数据的低维结构实现高效压缩与搜索。该方法对任意满足点间距离与存储编码所需内存成比例的距离函数均适用,包括字符串编辑距离(如Levenshtein、Needleman-Wunsch)及集合相异度量(如Jaccard、Dice)。我们在基因组、蛋白质组及集合数据等多种数据集上进行测试,对比了panCAKES与gzip的压缩率,以及压缩与未压缩版本的数据集间的搜索性能。结果表明,panCAKES的压缩率接近gzip,同时在k-NN和ρ-NN搜索中实现了亚线性时间复杂度。结论是:panCAKES是一种高效、通用的精确压缩搜索算法,适用于符合流形假设的大规模数据集。我们以Rust语言提供了panCAKES的开源实现。
原文摘要 · Abstract (English)
The Big Data explosion has necessitated the development of search algorithms that scale sub-linearly in time and memory. While compression algorithms and search algorithms do exist independently, few algorithms offer both, and those which do are domain-specific. We present panCAKES, a novel approach to compressive search, i.e., a way to perform $k$-NN and $ρ$-NN search on compressed data while only decompressing a small, relevant, portion of the data. panCAKES assumes the manifold hypothesis and leverages the low-dimensional structure of the data to compress and search it efficiently. panCAKES is generic over any distance function for which the distance between two points is proportional to the memory cost of storing an encoding of one in terms of the other. This property holds for many widely-used distance functions, e.g. string edit distances (Levenshtein, Needleman-Wunsch, etc.) and set dissimilarity measures (Jaccard, Dice, etc.). We benchmark panCAKES on a variety of datasets, including genomic, proteomic, and set data. We compare compression ratios to gzip, and search performance between the compressed and uncompressed versions of the same dataset. panCAKES achieves compression ratios close to those of gzip, while offering sub-linear time performance for $k$-NN and $ρ$-NN search. We conclude that panCAKES is an efficient, general-purpose algorithm for exact compressive search on large datasets that obey the manifold hypothesis. We provide an open-source implementation of panCAKES in the Rust programming language.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。