通过加权平均时间专家模型，高效实现跨时域泛化。
Scaling Up Temporal Domain Generalization via Temporal Experts Averaging
- 用约束微调构建功能多样但参数相似的时间专家模型
- 基于主成分空间的权重轨迹分析，自适应加权平均提升性能
- 在7个基准上比现有方法最高提升69%，效率高60倍
时间域泛化（TDG）旨在应对随时间变化的分布偏移，如词汇演变。以往方法多通过预测未来模型权重来实现，但全模型预测对中等规模模型而言成本过高。近期方法仅预测分类层，限制了其他组件的调整能力。为此，本文提出时间专家平均（TEA），一种新颖且可扩展的TDG框架：通过权重平均更新整个模型，在最大化泛化潜力的同时最小化计算开销。理论分析指导我们采用两步策略：首先，对无特定领域基础模型在各时间域上微调，并施加权重变化约束，生成功能多样但参数相似的专家模型；其次，在主成分子空间中建模时间权重轨迹，通过自适应平均系数优化偏差-方差权衡。专家贡献度基于其投影到未来域的接近程度。在7个TDG基准、5种模型和2种设置下实验表明，TEA相比之前方法性能最高提升69%，效率最高提升60倍。
原文摘要 · Abstract (English)
Temporal Domain Generalization (TDG) aims to generalize across temporal distribution shifts, e.g., lexical change over time. Prior work often addresses this by predicting future model weights. However, full model prediction is prohibitively expensive for even reasonably sized models. Thus, recent methods only predict the classifier layer, limiting generalization by failing to adjust other model components. To address this, we propose Temporal Experts Averaging (TEA), a novel and scalable TDG framework that updates the entire model using weight averaging to maximize generalization potential while minimizing computational costs. Our theoretical analysis guides us to two steps that enhance generalization to future domains. First, we create expert models with functional diversity yet parameter similarity by fine-tuning a domain-agnostic base model on individual temporal domains while constraining weight changes. Second, we optimize the bias-variance tradeoff through adaptive averaging coefficients derived from modeling temporal weight trajectories in a principal component subspace. Expert's contributions are based on their projected proximity to future domains. Extensive experiments across 7 TDG benchmarks, 5 models, and 2 TDG settings shows TEA outperforms prior TDG methods by up to 69% while being up to 60x more efficient.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。