用轻量稳定适配器让大模型高效预测时间序列。
One-for-All: A Lightweight Stabilized and Parameter-Efficient Pre-trained LLM for Time Series Forecasting
- 在位置编码和输出层注入低秩矩阵，冻结主干模型参数。
- 参数减少6.8到21倍，内存占用缩小168到1776倍，精度不降。
- 适合边缘设备部署，医疗、金融等实时场景可用。
针对预训练大语言模型在多变量时间序列分析中因计算与内存开销过大而难以部署的问题，本文提出One-for-All框架，引入高斯秩稳定低秩适配器（rsLoRA），实现冻结大模型的参数高效微调。rsLoRA基于数学推导的秩稳定性机制，确保低秩下的梯度稳定，是现有参数高效微调方法中未有之创新。该方法将可训练秩分解矩阵（秩16）注入位置嵌入和输出层，保持自注意力权重不变。相比TimesNet、GPT4TS和TIME-LLM，其可训练参数减少6.8×、21×和11.8×，内存占用降低至2.2MiB，远低于现有最优模型的340MiB–4.18GiB。在六个时间序列任务上的评估显示，One-for-All在效率-精度权衡上达到最优：参数效率比TimesNet高5.5×、比GPT4TS高21×，同时保持相同预测精度（MSE=0.33）。跨不同预测跨度（96–720步）和数据集（ETT、Weather、M3、M4）表现稳定，参数量仅为传统Transformer的1.7%。该成果使模型可在边缘设备上部署，适用于医疗、金融与环境监测场景。
原文摘要 · Abstract (English)
We address the challenge of adapting pre-trained Large Language Models (LLMs) for multivariate time-series analysis, where their deployment is often hindered by prohibitive computational and memory demands. Our solution, One-for-All, introduces Gaussian Rank-Stabilized Low-Rank Adapters (rsLoRA) to enable parameter-efficient fine-tuning of frozen LLMs. While inspired by LoRA, rsLoRA introduces a mathematically grounded rank-stabilization mechanism that enables provable gradient stability at low ranks a novel contribution absent in prior PEFT methods. Our framework injects trainable rank decomposition matrices (rank 16) into positional embeddings and output layers, while keeping self-attention weights fixed. This design reduces trainable parameters by 6.8$\times$ (vs. TimesNet), 21$\times$ (vs. GPT4TS), and 11.8$\times$ (vs. TIME-LLM), while achieving a 168-1,776$\times$ smaller memory footprint (2.2MiB vs. 340MiB-4.18GiB in SOTA models). Rigorous evaluation across six time-series tasks demonstrates that One-for-All achieves state-of-the-art efficiency-accuracy trade-offs: 5.5$\times$ higher parameter efficiency (MSE=5.50) than TimesNet and 21$\times$ better than GPT4TS, while matching their forecasting accuracy (MSE=0.33). The framework's stability is validated through consistent performance across diverse horizons (96-720 steps) and datasets (ETT, Weather, M3, M4), with 98.3% fewer parameters than conventional transformers. These advances enable deployment on edge devices for healthcare, finance, and environmental monitoring without compromising performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。