用张量分解实现高效微调，参数量更少且适应多任务。
MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuning
- 用共享张量分解因子化模型模块，降低参数冗余。
- 单任务与多任务下性能媲美顶尖方法，参数效率突出。
- 设计可自适应秩的优化器，提升训练收敛性。
我们提出MetaTT，一种基于张量分解（Tensor Train, TT）的预训练模型微调框架。通过使用单一共享的TT结构对Transformer子模块进行分解，该方法能灵活地捕捉层、矩阵类型等关键结构维度，并可选地整合注意力头和任务信息。这种设计使参数量随模式之和而非乘积增长，显著压缩适配器规模。在标准语言建模单任务基准测试中，MetaTT实现了与当前最优矩阵及张量分解方法相当的参数效率与精度平衡；在多任务学习中也表现优异。此外，我们受量子多体物理中的DMRG方法启发，设计了一种基于TT分解的秩自适应优化器，结合AdamW后显著提升了特定目标秩下的优化性能。
原文摘要 · Abstract (English)
We present MetaTT, a Tensor Train (TT) adapter framework for fine-tuning of pre-trained transformers. MetaTT enables flexible and parameter-efficient model adaptation by using a single shared TT to factorize transformer sub-modules. This factorization indexes key structural dimensions, including layer and matrix type, and can optionally incorporate heads and tasks. This design allows MetaTT's parameter count to scale with the sum, rather than the product, of the modes, resulting in a substantially more compact adapter. Our benchmarks compare MetaTT with LoRA along with recent state-of-the-art matrix and tensor decomposition based fine-tuning methods. We observe that when tested on single-task standard language modeling benchmarks, MetaTT achieves competitive parameter efficiency to accuracy tradeoff. We further demonstrate that MetaTT performs competitively when compared to state-of-the-art methods on multi-task learning. Finally, we leverage the TT decomposition to design a rank adaptive optimizer inspired by the DMRG method from many-body physics. Our results demonstrate that integrating this approach with AdamW enhances optimization performance for a specified target rank.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。