用500条高质量数据让小模型精准掌握专业领域知识
Domain Specific Specialization in Low-Resource Settings: The Efficacy of Offline Response-Based Knowledge Distillation in Large Language Models
- 用教师模型生成上下文相关的合成数据，替代大量原始文本
- 仅500行数据即达96.7%准确率，远超更大但无结构的数据集
- 适合资源有限却需高精度领域助手的场景
大型语言模型在通用任务上表现优异，但在缺乏预训练数据的领域知识任务中常出现幻觉。本文提出一种离线响应式知识蒸馏方法，在硬件受限条件下实现高精度专业化助手。评估了三种数据策略：通用领域适配（15,000行）、非结构化知识注入（2,000行）和由教师模型生成的上下文感知合成数据集（500行）。为降低计算成本，采用Unsloth库优化Qwen-2.5-7B学生模型，将NVIDIA A100 GPU内存需求从40 GB降至16 GB。实验表明，尽管更大规模的非结构化数据集仍存在持续幻觉问题，500行上下文感知数据集可达到96.7%准确率并具备强拒答能力。研究验证了LIMA假设，表明在低资源环境下，数据质量与结构对齐比数量更重要。
原文摘要 · Abstract (English)
Large Language Models (LLMs) excel in general tasks but often struggle with hallucinations when handling domain-specific or institutional knowledge absent from their pre-training. We present an offline response-based knowledge distillation method that develops high-accuracy specialized assistants under constrained hardware resources. We evaluate three distinct data strategies: general domain adaptation (15,000 lines), unstructured knowledge injection (2,000 lines), and a context-aware synthetic dataset (500 lines) generated by a teacher model. To minimize computational costs, we utilize the Unsloth library to optimize the Qwen-2.5-7B student model, reducing NVIDIA A100 GPU memory requirements from 40 GB to 16 GB. Experimental results demonstrate that while larger unstructured datasets suffer from persistent hallucinations, the 500-line context-aware dataset achieves a 96.7% accuracy rate and robust rejection capability. These findings validate the LIMA hypothesis, showing that data quality and structural alignment are more critical than quantity for domain adaptation in low-resource settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。