用KAN小模型协作,让大模型更省资源、更准、不遗忘。
KCM: KAN-Based Collaboration Models Enhance Pretrained Large Models
- 用KAN替代MLP做小模型,提升可解释性并缓解灾难性遗忘。
- 减少大模型调用次数,精度几乎不变,计算成本大幅下降。
- 特别适合长尾数据任务,对需要低资源高精度的场景友好。
近年来,预训练大模型(PLMs)研究提出了大-小模型协同框架,利用易于训练的小模型辅助大模型,旨在显著降低计算资源消耗的同时保持相近的准确率,并提升大模型在特定领域任务中的表现。然而,这种协同范式存在显著的准确率下降、灾难性遗忘加剧以及小模型知识引发的幻觉问题。为解决这些问题,我们提出一种基于KAN的协同模型(KCM),作为大-小模型协同的改进方案。KAN是一种区别于传统MLP的新型神经网络架构,相比MLP具有更优的可视化与可解释性,并能有效缓解灾难性遗忘。我们在语言、视觉及跨模态视觉-语言三类任务中部署KCM,实验结果表明:相较于纯大模型方法,采用KCM的协同框架显著减少了大模型推理调用次数,同时保持近似相同的任务准确率,从而大幅降低计算资源消耗;同时,基于KAN的小协同模型显著缓解了灾难性遗忘,在长尾数据上带来明显准确率提升。结果显示,与基于MLP的小协同模型(MCM)相比,KCM在所有指标上均表现更优。
原文摘要 · Abstract (English)
In recent years, Pretrained Large Models(PLMs) researchers proposed large-small model collaboration frameworks, leveraged easily trainable small models to assist large models, aim to(1) significantly reduce computational resource consumption while maintaining comparable accuracy, and (2) enhance large model performance in specialized domain tasks. However, this collaborative paradigm suffers from issues such as significant accuracy degradation, exacerbated catastrophic forgetting, and amplified hallucination problems induced by small model knowledge. To address these challenges, we propose a KAN-based Collaborative Model (KCM) as an improved approach to large-small model collaboration. The KAN utilized in KCM represents an alternative neural network architecture distinct from conventional MLPs. Compared to MLPs, KAN offers superior visualizability and interpretability while mitigating catastrophic forgetting. We deployed KCM in large-small model collaborative systems across three scenarios: language, vision, and vision-language cross-modal tasks. The experimental results demonstrate that, compared with pure large model approaches, the large-small model collaboration framework utilizing KCM as the collaborative model significantly reduces the number of large model inference calls while maintaining near-identical task accuracy, thereby substantially lowering computational resource consumption. Concurrently, the KAN-based small collaborative model markedly mitigates catastrophic forgetting, leading to significant accuracy improvements for long-tail data. The results reveal that KCM demonstrates superior performance across all metrics compared to MLP-based small collaborative models (MCM).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。