通过数据与模型压缩，20倍加速选最优大模型微调。
DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression
- 按数据分布、质量分层筛选并压缩关键词，提升训练效率。
- 基于层重要性评分剪枝，结合稀疏合并保留模型能力。
- 适用场景：医疗、金融等领域的高效模型选型与微调。
大型语言模型在通用任务中表现优异，但在领域特定任务中需微调特定数据。面对众多开源LLM，如何快速识别最优微调模型成为难题。本文提出数据与模型压缩框架DaMoC：在数据层面，系统构建三类数据筛选范式——分布感知、质量感知及混合方法，并通过增强关键标记密度实现文本压缩，再利用大模型迭代重写优化表达；在模型层面，基于层相似性评分评估各层重要性并剔除低重要层，引入稀疏合并策略以最大程度保留原始模型能力。在医疗问答、金融问答、通用问答和阅读理解四个数据集上的大量实验表明，该方法可在节省约20倍训练时间的前提下选出最优模型。
原文摘要 · Abstract (English)
Large language models (LLMs) excel in general tasks but struggle with domain-specific ones, requiring fine-tuning with specific data. With many open-source LLMs available, selecting the best model for fine-tuning downstream tasks is challenging, primarily focusing on how to quickly identify the optimal LLM. We introduce a Data and Model Compression Framework (DaMoC) that addresses this challenge by: 1) Data Level: A systematic categorization of data filtering methodologies for LLMs is first established, classifying them into three distinct paradigms: (1) distribution-aware methods, (2) quality-aware methods, and (3) hybrid approaches considering both dimensions. Further, we enhance the density of key tokens in the text achieving token compression. Subsequently, we use an LLM to iterative rewrite the text to optimize its expression. 2) Model Level: We use layer similarity scores to assess each layer's importance and remove those with lower importance. Then, we introduce a sparse merging paradigm to preserve as much of the original model's capability as possible. Extensive experiments on four datasets, medical Q&A, financial Q&A, general Q&A, and reading comprehension, show that we can select the optimal LLM while saving approximately 20-fold in training time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。