用动态路由和置信度门控,从不匹配的预训练模型中提炼轻量预测知识。
When to Trust, How to Distill: Multi-Foundation Model Guidance for Lightweight, Robust Scientific Time Series Forecasting

- 根据输入局部特征动态选最优教师,融合多模型互补优势。
- 当教师预测与实际偏差大时自动降低蒸馏强度,避免错误传递。
- 在气象、碳通量等4个科学领域验证,边缘设备部署精度显著提升。
在物理科学中部署时间序列基础模型(TSFMs)面临核心矛盾:尽管其蕴含丰富的通用时序规律,但在零样本迁移至特定科学领域时存在严重分布偏移,且计算开销大,难以在边缘传感网络中应用。本文提出一种新框架Guard,将多教师蒸馏重构为实例级决策过程,包含两个自适应机制:(1) 上下文路由器,基于输入局部统计量动态选择最相关教师,利用不同基础模型间的互补性;(2) 不确定性门控温度机制,作为“断路器”,在教师置信度偏离领域真实情况时自动减弱蒸馏强度。在四个气候关键领域(气象学、生态系统碳通量、土壤湿度、能源电网)上评估,本方法显著降低相对于固定权重多教师蒸馏基线的均方根误差,成功从存在分布偏移的预训练教师模型中蒸馏出有效知识。即使教师在目标域上零样本表现不佳,仍可作为关键修正项,在最难的28.5%实例上优于全局最优基础模型。最终实现适用于资源受限边缘部署的高精度科学预测。代码已公开于https://github.com/RupasreeDey/GUARD-KDD2026。
原文摘要 · Abstract (English)
The deployment of Time-Series Foundation Models (TSFMs) in physical sciences is hindered by a critical trade-off: while these models encode rich, universal temporal dynamics, they suffer from severe distributional misalignment when applied zero-shot to specific scientific domains, and their computational cost prohibits deployment in edge-computing sensor networks. We address a fundamental challenge: How can we extract latent structural knowledge from misaligned foundation models (FM) to train lightweight, specialized forecasters? We propose Gated Uncertainty-Aware Routing for Distillation (Guard), a novel framework that reframes multiteacher distillation as an instance-wise decision process with two adaptive mechanisms: (1) a Contextual Router that dynamically selects the most relevant teacher based on local input statistics, exploiting complementarity across diverse foundation models; and (2) an Uncertainty-Gated Temperature mechanism that acts as a "circuit-breaker," automatically attenuating distillation strength when teacher confidence diverges from domain reality. We evaluate our proposed lightweight framework on four climate-critical domains: meteorology, ecosystem carbon flux, soil moisture, and energy grids. Our method significantly reduces RMSE relative to a fixed-weight multi-teacher distillation baseline, successfully distilling knowledge from pretrained FMs (teachers) even when they exhibit suboptimal zero-shot accuracy due to distribution shift between the original and target data domains. We demonstrate that these domain-misaligned teachers can still serve as critical correctives, outperforming the globally superior FMs on 28.5% of the hardest instances. Ultimately, this enables high-precision scientific forecasting suitable for resource-constrained edge deployment. Code is available at https://github.com/RupasreeDey/GUARD-KDD2026.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。