用元认知知识和分组偏好优化,缓解小模型微调时的灾难性遗忘。
MetaGDPO: Alleviating Catastrophic Forgetting with Metacognitive Knowledge through Group Direct Preference Optimization
- 构建含元认知标注的5000条推理数据集,匹配任务与模型能力。
- 提出GDPO方法,提升小模型在资源受限下的知识保留与性能。
- 适合关注小模型压缩与持续学习的研究者或工程师。
大型语言模型具备强大的推理能力,可有效压缩为小型模型。然而,现有数据集和微调方法仍存在灾难性遗忘问题,尤其在模型小于80亿参数时更为明显。首先,多数数据集忽略训练数据知识与模型固有能力的关系,难以保留先验知识;其次,传统训练目标无法有效约束内在知识的保留,导致先前技能丢失。为此,本文从数据与微调方法两方面提出综合性解决方案。在数据层面,构建包含5000个实例的数据集,覆盖多种推理任务并融入元认知知识,提升对小型模型的知识蒸馏效果。通过标注每道题所需的元认知知识,并基于任务知识与模型能力进行数据筛选。在训练层面,提出GDPO(Group Direction Preference Optimization),更适合资源受限场景,能高效逼近GRPO性能。在大模型引导下,通过参考模型隐式约束优化路径,实现更有效的知识迁移并抑制参数过度漂移。大量实验表明,该方法显著缓解灾难性遗忘,提升小模型的推理表现。
原文摘要 · Abstract (English)
Large Language Models demonstrate strong reasoning capabilities, which can be effectively compressed into smaller models. However, existing datasets and fine-tuning approaches still face challenges that lead to catastrophic forgetting, particularly for models smaller than 8B. First, most datasets typically ignore the relationship between training data knowledge and the model's inherent abilities, making it difficult to preserve prior knowledge. Second, conventional training objectives often fail to constrain inherent knowledge preservation, which can result in forgetting of previously learned skills. To address these issues, we propose a comprehensive solution that alleviates catastrophic forgetting from both the data and fine-tuning approach perspectives. On the data side, we construct a dataset of 5K instances that covers multiple reasoning tasks and incorporates metacognitive knowledge, making it more tolerant and effective for distillation into smaller models. We annotate the metacognitive knowledge required to solve each question and filter the data based on task knowledge and the model's inherent skills. On the training side, we introduce GDPO (Group Direction Preference Optimization), which is better suited for resource-limited scenarios and can efficiently approximate the performance of GRPO. Guided by the large model and by implicitly constraining the optimization path through a reference model, GDPO enables more effective knowledge transfer from the large model and constrains excessive parameter drift. Extensive experiments demonstrate that our approach significantly alleviates catastrophic forgetting and improves reasoning performance on smaller models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。