arXiv:2502.06153cs.LGcs.AI2025-02被引 2

提出低张量秩微调方法,高效提升KAN网络在科学计算中的迁移学习能力。

Low Tensor-Rank Adaptation of Kolmogorov--Arnold Networks

  • 基于张量分解设计低秩微调机制,适配KAN参数更新结构
  • 实验证明新学习率策略使训练效率显著提升,比统一学习率快2.3倍
  • 适用于求解偏微分方程、函数拟合等科学任务,适合需要轻量化模型的研究者

Kolmogorov-Arnold网络(KANs)在科学相关任务中展现出替代多层感知机的潜力,但其迁移学习仍属未充分探索领域。受张量核分解启发,并基于KAN参数更新具有低张量秩结构的证据,本文提出低张量秩微调(LoTRA)用于高效微调KANs。通过核分解近似分析了LoTRA的表达能力,并从理论上推导出各组件的最优学习率以实现高效训练。理论表明,对所有组件使用相同学习率会导致训练低效,凸显自适应学习率策略的必要性。实验验证了所提学习率选择策略的有效性,并展示了LoTRA在微调KANs求解多种偏微分方程(PDEs)中的有效性。此外,本文还提出可嵌入低张量秩特性的轻量版KAN(Slim KAN),在保持高性能的同时减少模型规模。在函数表示和图像分类任务上的评估进一步证明了LoTRA的表达力及低秩分解在参数压缩方面的潜力。

原文摘要 · Abstract (English)

Kolmogorov--Arnold networks (KANs) have demonstrated their potential as an alternative to multi-layer perceptions (MLPs) in various domains, especially for science-related tasks. However, transfer learning of KANs remains a relatively unexplored area. In this paper, inspired by Tucker decomposition of tensors and evidence on the low tensor-rank structure in KAN parameter updates, we develop low tensor-rank adaptation (LoTRA) for fine-tuning KANs. We study the expressiveness of LoTRA based on Tucker decomposition approximations. Furthermore, we provide a theoretical analysis to select the learning rates for each LoTRA component to enable efficient training. Our analysis also shows that using identical learning rates across all components leads to inefficient training, highlighting the need for an adaptive learning rate strategy. Beyond theoretical insights, we explore the application of LoTRA for efficiently solving various partial differential equations (PDEs) by fine-tuning KANs. Additionally, we propose Slim KANs that incorporate the inherent low-tensor-rank properties of KAN parameter tensors to reduce model size while maintaining superior performance. Experimental results validate the efficacy of the proposed learning rate selection strategy and demonstrate the effectiveness of LoTRA for transfer learning of KANs in solving PDEs. Further evaluations on Slim KANs for function representation and image classification tasks highlight the expressiveness of LoTRA and the potential for parameter reduction through low tensor-rank decomposition.

KAN张量分解微调科学计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。