用知识蒸馏融合多种时间先验,提升Transformer在非平稳金融时序中的预测能力。
Integrating Inductive Biases in Transformers via Distillation for Financial Time Series Forecasting
- 通过注意力掩码训练多个具特定先验的教师模型,再蒸馏到统一学生模型。
- 在四大股市上超越强基线模型,年化收益提升55%,推理计算量仅需38%。
- 能自适应不同市场阶段,行为匹配经典模型的盈利周期,适合金融时序预测场景。
基于Transformer的模型因强大的表征能力和灵活架构被广泛用于时序预测,但其常隐含平稳性与稳定动态假设,这在存在制度切换和非平稳性的金融市场中不成立。实证表明,最先进的时序Transformer在金融任务上甚至不如基础Transformer,而具有明确归纳偏置的简单结构如CNN和RNN反而表现更优。单一偏置无法在所有市场或周期中占优,因此稳健的金融预测需融合互补的时间先验。本文提出TIPS(Transformer with Inductive Prior Synthesis),一种通过知识蒸馏整合因果性、局部性和周期性等多种归纳偏置的框架。TIPS通过注意力掩码训练具备特定偏置的Transformer教师模型,并将它们的知识蒸馏到一个学生模型中,实现不同制度下的偏置自适应对齐。在四个主要股票市场中,TIPS实现了当前最优性能,年化收益率、夏普比率和卡尔玛比率分别较强基线提升55%、9%和16%,同时仅需38%的推理计算量。进一步分析显示,TIPS产生的超额收益具有统计显著性,且在经典模型盈利期表现出与之相符的动态行为。结果强调了在非平稳金融时序中,制度依赖的归纳偏置利用对泛化鲁棒性至关重要。
原文摘要 · Abstract (English)
Transformer-based models have been widely adopted for time-series forecasting due to their high representational capacity and architectural flexibility. However, many Transformer variants implicitly assume stationarity and stable temporal dynamics -- assumptions routinely violated in financial markets characterized by regime shifts and non-stationarity. Empirically, state-of-the-art time-series Transformers often underperform even vanilla Transformers on financial tasks, while simpler architectures with distinct inductive biases, such as CNNs and RNNs, can achieve stronger performance with substantially lower complexity. At the same time, no single inductive bias dominates across markets or regimes, suggesting that robust financial forecasting requires integrating complementary temporal priors. We propose TIPS (Transformer with Inductive Prior Synthesis), a knowledge distillation framework that synthesizes diverse inductive biases -- causality, locality, and periodicity -- within a unified Transformer. TIPS trains bias-specialized Transformer teachers via attention masking, then distills their knowledge into a single student model with regime-dependent alignment across inductive biases. Across four major equity markets, TIPS achieves state-of-the-art performance, outperforming strong ensemble baselines by 55%, 9%, and 16% in annual return, Sharpe ratio, and Calmar ratio, while requiring only 38% of the inference-time computation. Further analyses show that TIPS generates statistically significant excess returns beyond both vanilla Transformers and its teacher ensembles, and exhibits regime-dependent behavioral alignment with classical architectures during their profitable periods. These results highlight the importance of regime-dependent inductive bias utilization for robust generalization in non-stationary financial time series.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。