提出PMKLC压缩框架,大幅提升基因组数据压缩率与速度。
PMKLC: Parallel Multi-Knowledge Learning-based Lossless Compression for Large-Scale Genomics Database
- 基于多知识学习的并行压缩框架,提升压缩率和鲁棒性。
- 单/多GPU版本分别实现3.0倍和10.7倍吞吐加速,压缩率提升超73%。
- 适合资源受限设备与大规模基因组数据场景,稳定高效。
基于学习的无损压缩器在大规模基因组数据库备份、存储、传输与管理中至关重要。然而,现有方法普遍存在压缩率不足、压缩/解压吞吐低以及压缩鲁棒性差的问题,限制了其在产业界和学术界的广泛应用。为此,本文提出一种新型并行多知识学习无损压缩框架PMKLC,包含四大设计:1)构建自动化多知识学习压缩框架作为核心,提升压缩率与鲁棒性;2)设计基于GPU加速的($s$,$k$)-mer编码器,优化吞吐与资源利用;3)引入数据块分割与分步模型传递(SMP)机制实现并行加速;4)设计PMKLC-S(单GPU)与PMKLC-M(多GPU)两种模式以适应复杂应用场景。我们在15个不同物种、不同规模的真实数据集上对PMKLC-S/M与14种基线(7种传统、7种学习型)进行了评测。相比基线,PMKLC-S/M平均压缩率提升最高达73.609%和73.480%,平均吞吐提升最高达3.036×和10.710×。此外,PMKLC-S/M在抗分布扰动方面表现最佳,且内存开销具有竞争力,展现出在资源受限设备上的强适应能力。
原文摘要 · Abstract (English)
Learning-based lossless compressors play a crucial role in large-scale genomic database backup, storage, transmission, and management. However, their 1) inadequate compression ratio, 2) low compression \& decompression throughput, and 3) poor compression robustness limit their widespread adoption and application in both industry and academia. To solve those challenges, we propose a novel \underline{P}arallel \underline{M}ulti-\underline{K}nowledge \underline{L}earning-based \underline{C}ompressor (PMKLC) with four crucial designs: 1) We propose an automated multi-knowledge learning-based compression framework as compressors' backbone to enhance compression ratio and robustness; 2) we design a GPU-accelerated ($s$,$k$)-mer encoder to optimize compression throughput and computing resource usage; 3) we introduce data block partitioning and Step-wise Model Passing (SMP) mechanisms for parallel acceleration; 4) We design two compression modes PMKLC-S and PMKLC-M to meet the complex application scenarios, where the former runs on a resource-constrained single GPU and the latter is multi-GPU accelerated. We benchmark PMKLC-S/M and 14 baselines (7 traditional and 7 leaning-based) on 15 real-world datasets with different species and data sizes. Compared to baselines on the testing datasets, PMKLC-S/M achieve the average compression ratio improvement up to 73.609\% and 73.480\%, the average throughput improvement up to 3.036$\times$ and 10.710$\times$, respectively. Besides, PMKLC-S/M also achieve the best robustness and competitive memory cost, indicating its greater stability against datasets with different probability distribution perturbations, and its strong ability to run on memory-constrained devices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。