arXiv:2410.02147cs.LGcs.AI2024-10ICLR被引 6

通过分解模型参数,实现时间序列无源域适应的高效迁移。

Efficient Source-Free Time-Series Adaptation via Parameter Subspace Disentanglement

  • 将源模型权重按张量分解方式重参数化,压缩模型规模。
  • 仅微调部分分解因子,使可训练参数减少90%以上。
  • 兼容多种方法,适合资源受限设备部署。

本文提出一种面向时间序列的高效无源域适应(SFDA)框架,旨在提升参数效率与数据利用效率。通过在源模型准备阶段采用类似张量分解的重参数化方式,将模型压缩为紧凑形式;目标域适应时仅微调部分分解因子,显著提升训练效率。我们基于PAC贝叶斯分析证明,这种选择性微调策略通过限制模型学习能力,隐式起到正则化作用。该方法不仅降低整体模型规模,提升推理效率,还兼容多种现有SFDA方法,在保持性能的同时,使可训练参数数量和推理计算量(MACs)均减少超过90%。

原文摘要 · Abstract (English)

In this paper, we propose a framework for efficient Source-Free Domain Adaptation (SFDA) in the context of time-series, focusing on enhancing both parameter efficiency and data-sample utilization. Our approach introduces an improved paradigm for source-model preparation and target-side adaptation, aiming to enhance training efficiency during target adaptation. Specifically, we reparameterize the source model's weights in a Tucker-style decomposed manner, factorizing the model into a compact form during the source model preparation phase. During target-side adaptation, only a subset of these decomposed factors is fine-tuned, leading to significant improvements in training efficiency. We demonstrate using PAC Bayesian analysis that this selective fine-tuning strategy implicitly regularizes the adaptation process by constraining the model's learning capacity. Furthermore, this re-parameterization reduces the overall model size and enhances inference efficiency, making the approach particularly well suited for resource-constrained devices. Additionally, we demonstrate that our framework is compatible with various SFDA methods and achieves significant computational efficiency, reducing the number of fine-tuned parameters and inference overhead in terms of MACs by over 90% while maintaining model performance.

时间序列域适应参数效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。