arXiv:2511.15977cs.DCcs.AI2025-11AAAI

通过自适应调度优化基因组分析内存使用,提升大规模基因组流程效率

Efficient Chromosome Parallelization for Precision Medicine Genomic Workflows

  • 基于符号回归预测染色体级内存需求,避免过度分配
  • 动态调度将任务按内存需求分批处理,减少内存溢出
  • 适合需要高效运行大样本基因组分析的研究人员

精准医学中的大规模基因组工作流每样本数据量可达数十至数百吉字节,导致内存峰值高、磁盘I/O密集,常因内存不足失败。传统静态资源分配难以应对染色体级别内存需求的波动,造成资源利用率低、运行时间长。本文提出多种自适应、内存高效的染色体级并行化机制:首先构建符号回归模型,估算各染色体任务的内存消耗,并引入插值偏差以保守性地最小化过分配;其次设计动态调度器,采用多项式回归预测内存使用,将任务打包视为背包问题,依据预测内存需求最优批次执行;此外,提出静态调度器,通过优化染色体处理顺序,在不降低吞吐量的前提下最小化峰值内存。在模拟与真实基因组流水线上的评估表明,所提方法有效减少内存溢出,均衡线程负载,显著缩短端到端执行时间。

原文摘要 · Abstract (English)

Large-scale genomic workflows used in precision medicine can process datasets spanning tens to hundreds of gigabytes per sample, leading to high memory spikes, intensive disk I/O, and task failures due to out-of-memory errors. Simple static resource allocation methods struggle to handle the variability in per-chromosome RAM demands, resulting in poor resource utilization and long runtimes. In this work, we propose multiple mechanisms for adaptive, RAM-efficient parallelization of chromosome-level bioinformatics workflows. First, we develop a symbolic regression model that estimates per-chromosome memory consumption for a given task and introduces an interpolating bias to conservatively minimize over-allocation. Second, we present a dynamic scheduler that adaptively predicts RAM usage with a polynomial regression model, treating task packing as a Knapsack problem to optimally batch jobs based on predicted memory requirements. Additionally, we present a static scheduler that optimizes chromosome processing order to minimize peak memory while preserving throughput. Our proposed methods, evaluated on simulations and real-world genomic pipelines, provide new mechanisms to reduce memory overruns and balance load across threads. We thereby achieve faster end-to-end execution, showcasing the potential to optimize large-scale genomic workflows.

基因组分析内存优化并行调度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。