通过剔除异常值，实现高效低耗的基因组大模型训练与推理。
Fast and Low-Cost Genomic Foundation Models via Outlier Removal
- 用类记忆关联机制替代原注意力层，剔除训练与微调中的异常数据
- 微调性能提升37.98%，量化效果提升64.34%，计算成本显著降低
- 适合算力有限但需高性能基因组建模的研究者使用
为应对基因组建模中计算资源稀缺的问题，我们提出GERM，一种具备强压缩性能和快速适配能力的基因组基础模型。GERM通过剔除阻碍低秩微调和后训练量化的异常值，提升了效率与鲁棒性。其采用受联想记忆模型启发的无异常值机制替代原始注意力层。在预训练与微调阶段均剔除异常值，显著加速适应过程，降低计算开销，并增强量化鲁棒性，损失在可接受范围内。此外，提出GERM-T策略，在无异常值框架内采用小步持续学习，利用原始检查点避免从头训练。实验表明，GERM相比基线模型，微调性能提升37.98%，量化性能提升64.34%，平均峰度降低92.14%，最大无穷范数下降82.77%。相较于领先方法，GERM始终表现更优，为资源受限环境下的基因组建模提供实用方案。代码已开源：https://github.com/MAGICS-LAB/GERM。
原文摘要 · Abstract (English)
To address the challenge of scarce computational resources in genomic modeling, we introduce GERM, a genomic foundation model with strong compression performance and fast adaptability. GERM improves upon models like DNABERT-2 by eliminating outliers that hinder low-rank adaptation and post-training quantization, enhancing both efficiency and robustness. We replace the vanilla attention layer with an outlier-free mechanism inspired by associative memory models. By removing outliers during both pre-training and fine-tuning, this approach accelerates adaptation, reduces computational costs, and enhances quantization robustness within acceptable loss margins. Additionally, we propose GERM-T, a strategy that employs small-step continual learning within the outlier-free framework, leveraging original checkpoints to avoid retraining from scratch. Empirically, GERM improves fine-tuning performance by 37.98% and quantization by 64.34% over the baseline model. It also reduces average kurtosis by 92.14% and maximum infinity norm by 82.77%. Compared to leading methods, GERM consistently delivers superior performance, offering a practical solution for genomic modeling in resource-constrained settings. Code is available at https://github.com/MAGICS-LAB/GERM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。