用通用数据提升低资源领域模型性能,效果远超只用领域数据。
Mining Useful General Data for Low-Resource Domain Adaptation
- 基于神经正切核思想,自动筛选对领域适配最有帮助的通用数据。
- 在多个领域实验中,相比仅用领域数据提升8.7和5.1分。
- 适合数据稀缺但有通用数据可用的场景,如医疗、金融等。
将大语言模型(LLMs)适配到低资源领域仍具挑战性,因领域特定数据稀缺。尽管领域内数据有限,但存在大量与领域任务具有相似问答格式和推理模式的通用领域数据。这引发关键问题:能否从通用数据中挖掘出有助于低资源领域适配的有效信息?初步发现表明,无需精细筛选,通用链式思维数据就包含对领域适配有用的辅助信号。这一观察催生了一种超越依赖领域特定数据的新范式。为系统识别最有益的通用样本,提出NTK-Selector,其灵感来自神经正切核(NTK)捕捉训练动态对齐的能力。由于直接应用NTK于预训练LLM不现实,引入无雅可比矩阵的NTK近似,并实证证明其在微调过程中表现出稳定的类似NTK行为。跨医学、金融、法律和心理领域的广泛实验表明,NTK-Selector持续优于仅使用领域数据的微调及现有数据选择基线。尤其在Llama3-8B-Instruct和Qwen3-8B上,分别获得+8.7和+5.1的提升,而仅用领域数据的提升仅为+0.8和+0.9。
原文摘要 · Abstract (English)
Adapting large language models (LLMs) to low-resource domains remains challenging due to the scarcity of domain-specific data. While in-domain data is limited, there exists a vast amount of general-domain data that shares similar question-answer formats and reasoning patterns with domain tasks. This observation raises an important question: can useful general-domain data be mined to improve low-resource domain adaptation? Our initial findings show that general-domain chain-of-thought data contains useful auxiliary signals for domain adaptation, even without careful selection. This observation motivates a new paradigm for domain adaptation beyond exclusive reliance on domain-specific data. To systematically identify the most beneficial general-domain samples, we propose NTK-Selector, motivated by the Neural Tangent Kernel's ability to capture alignment in training dynamics. Since directly applying NTK to pretrained LLMs is impractical, we introduce a Jacobian-free NTK approximation and empirically demonstrate stable NTK-like behavior during fine-tuning. Extensive experiments across medical, financial, legal, and psychological domains demonstrate that NTK-Selector consistently outperforms domain-only fine-tuning and existing data selection baselines. In particular, NTK-Selector achieves gains of +8.7 and +5.1 points on Llama3-8B-Instruct and Qwen3-8B, respectively, compared to only +0.8 and +0.9 points from domain-only fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。