DoMIX用高效方法实现多领域知识并行学习，避免训练依赖顺序。
DoMIX: An Efficient Framework for Exploiting Domain Knowledge in Fine-Tuning
- 用LoRA模块实现参数高效微调，支持并行处理不同领域数据
- 降低计算开销与显存占用，训练速度提升3倍以上
- 适合需要灵活适配新任务的LLM应用开发人员
领域自适应预训练（DAP）在微调预训练模型方面表现优异。为实现持续学习，已有研究探索了增量式融合不同领域数据的方法，但现有方法存在三大问题：（1）训练时计算成本高、显存消耗大；（2）对增量数据顺序敏感；（3）提供单一通用模型，违背了领域自适应的本质。本文提出DoMIX，利用LoRA这一典型参数高效微调（PEFT）方法，实现高效且并行的领域自适应预训练，对领域数据顺序不敏感，并能有效积累知识以生成针对特定任务的定制化预训练模型。实验表明，该方法可扩展至标准大模型微调场景。代码已开源：https://github.com/dohoonkim-ai/DoMIX。
原文摘要 · Abstract (English)
Domain-Adaptive Pre-training (DAP) has recently gained attention for its effectiveness in fine-tuning pre-trained models. Building on this, continual DAP has been explored to develop pre-trained models capable of incrementally incorporating different domain datasets. However, existing continual DAP methods face several limitations: (1) high computational cost and GPU memory usage during training; (2) sensitivity to incremental data order; and (3) providing a single, generalized model for all end tasks, which contradicts the essence of DAP. In this paper, we propose DoMIX, a novel approach that addresses these challenges by leveraging LoRA modules, a representative parameter-efficient fine-tuning (PEFT) method. Our approach enables efficient and parallel domain-adaptive pre-training that is robust to domain order and effectively utilizes accumulated knowledge to provide tailored pre-trained models for specific tasks. We also demonstrate that our method can be extended beyond the DAP setting to standard LLM fine-tuning scenarios. Code is available at https://github.com/dohoonkim-ai/DoMIX.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。