用图像压缩思路重构基因数据格式,存储减半且无需解压
FASTR: Reimagining FASTQ via Compact Image-inspired Representation
- 将碱基和质量分数合并为8位编码,直接减少存储体积
- 相比FASTQ压缩快2.47至4.8倍,解压快1.75至2.34倍
- 可直接作为向量或图像输入模型,适合机器学习应用
高通量测序(HTS)虽推动群体基因组学发展,但产生海量数据,造成存储、传输与分析瓶颈。标准格式FASTQ每碱基占用1字节,每质量值1字节,导致输入输出效率低、成本高且冗余严重。现有压缩工具虽缓解部分问题,但常引入昂贵的解压开销或复杂依赖。本文提出FASTR,一种无损、计算原生的FASTQ替代方案,将每个核苷酸及其质量评分合并为单个8位值,文件大小至少减少2倍,且完全可逆,可直接用于下游分析。在Illumina、HiFi和ONT数据上,对FASTR使用通用压缩工具,压缩速度提升2.47至4.8倍,解压速度提升1.75至2.34倍。FASTR支持机器学习,读段可直接作为数值向量或图像形式输入。我们提供高度并行的转换软件生态,并验证其与minimap2等工具兼容,接口改动极小,无性能损耗。通过消除解压开销和减少数据移动,FASTR为可扩展基因组分析与实时测序工作流奠定基础。
原文摘要 · Abstract (English)
Motivation: High-throughput sequencing (HTS) enables population-scale genomics but generates massive datasets, creating bottlenecks in storage, transfer, and analysis. FASTQ, the standard format for over two decades, stores one byte per base and one byte per quality score, leading to inefficient I/O, high storage costs, and redundancy. Existing compression tools can mitigate some issues, but often introduce costly decompression or complex dependency issues. Results: We introduce FASTR, a lossless, computation-native successor to FASTQ that encodes each nucleotide together with its base quality score into a single 8-bit value. FASTR reduces file size by at least 2x while remaining fully reversible and directly usable for downstream analyses. Applying general-purpose compression tools on FASTR consistently yields higher compression ratios, 2.47, 3.64, and 4.8x faster compression, and 2.34, 1.96, 1.75x faster decompression than on FASTQ across Illumina, HiFi, and ONT reads. FASTR is machine-learning-ready, allowing reads to be consumed directly as numerical vectors or image-like representations. We provide a highly parallel software ecosystem for FASTQ-FASTR conversion and show that FASTR integrates with existing tools, such as minimap2, with minimal interface changes and no performance overhead. By eliminating decompression costs and reducing data movement, FASTR lays the foundation for scalable genomics analyses and real-time sequencing workflows. Availability and Implementation: https://github.com/ALSER-Lab/FASTR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。