GeneZip通过基因结构引导压缩，实现长段DNA建模的高效与精准。
GeneZip: Region-Aware Compression for Long Context DNA Modeling
- 基于基因区域信息动态分配每令牌碱基数，实现分区域智能压缩。
- 在4项任务中平均排名最优，137.6 BPT下仍保持最佳验证困惑度。
- 可处理128K上下文，微调速度比JanusDNA快50倍，适合大模型训练。
长上下文DNA模型受限于令牌混合开销及压缩对基因组表示预算的分配方式。现有方法多在碱基对层面操作，采用固定下采样或学习内容相关片段但无显式基因组预算，导致预训练成本高且难控制。本文提出GeneZip，一种结合H-Net式动态路由与区域感知比率（RAR）目标的压缩框架，利用静态基因结构注释设定各区域每令牌碱基数（BPT）目标；推理时无需注释即可压缩未见原始DNA。GeneZip有三大优势：其一有效——变体在编码器类压缩器中表现最佳，如GeneZip-70M在137.6 BPT下表现优异，并在四个可复现的DNALongBench任务（接触图预测、eQTL预测、增强子-靶基因预测、转录起始信号预测）中取得最高平均排名；其二具备冗余感知能力——后验分析显示，未经重复序列监督，GeneZip仍为转座子衍生重复和串联重复等主要重复序列分配更高局部BPT；其三高效——通过减少有效令牌混合长度，支持更长上下文与更大容量预训练，可在单张A100 80GB GPU上实现128K上下文与636M参数模型，且eQTL任务微调速度比JanusDNA快50.4倍（50对2520分钟）。这些结果确立GeneZip作为长上下文DNA建模的有效、冗余感知、高效压缩接口。
原文摘要 · Abstract (English)
Long-context DNA models are limited by token-mixing cost and by how compression allocates representational budget across the genome. Existing approaches operate close to base-pair resolution, apply fixed downsampling, or learn content-dependent chunks without an explicit genomic budget, making long-context pretraining expensive and difficult to control. We introduce GeneZip, a region-aware DNA compression framework that combines H-Net-style dynamic routing with a Region-Aware Ratio (RAR) objective and bounded routing. GeneZip uses static gene-structure annotations during compression training to specify region-wise base-pairs-per-token (BPT) targets; at inference time, it compresses raw unseen DNA without annotations. GeneZip provides three main benefits. First, it is effective: GeneZip variants achieve the best validation PPL among encoder-based compressors, with GeneZip-70M operating at 137.6 BPT, and across four reproducible DNALongBench tasks--contact map prediction, eQTL prediction, enhancer-target gene prediction, and transcription-initiation signal prediction--GeneZip obtains the best average rank among compared sequence models. Second, it is redundancy-aware: a post-hoc RepeatMasker/TRF analysis shows that, without repeat supervision, GeneZip assigns higher local BPT to TE-derived interspersed repeats and tandem repeats, two major classes of repetitive DNA sequence redundancy. Third, it is efficient: by reducing the effective token-mixing length, GeneZip enables longer-context and larger-capacity pretraining, including 128K-context and 636M-parameter variants on a single A100 80GB GPU, and fine-tunes the eQTL task 50.4x faster than JanusDNA (50 vs. 2520 minutes). These results establish GeneZip as an effective, redundancy-aware, and efficient compression interface for long-context DNA modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。