通过软路由机制让专家模型分工，减少冗余特征学习。
Soft Task-Aware Routing of Experts for Equivariant Representation Learning
- 将投影头视为专家，用软路由实现任务感知分工
- 降低不变与等变嵌入间的典型相关性，减少信息冗余
- 适用于需联合学习对称与不变特征的迁移学习场景
等变表示学习旨在捕捉输入变换在表示空间中的变化，而不变表示学习则忽略这些变换以编码语义信息。近期研究表明，联合学习两种表示通常有助于下游任务，通常通过独立的投影头实现。然而，这种设计忽略了不变与等变学习之间的共享信息，导致冗余特征学习和模型容量利用效率低下。为此，我们提出软任务感知路由（STAR），将投影头建模为专家。STAR使专家专长于捕获共享或任务特定信息，从而减少冗余特征学习。通过观察不变与等变嵌入间较低的典型相关性验证了该效果。实验结果表明，在多种迁移学习任务中均取得一致提升。代码已开源：https://github.com/YonseiML/star。
原文摘要 · Abstract (English)
Equivariant representation learning aims to capture variations induced by input transformations in the representation space, whereas invariant representation learning encodes semantic information by disregarding such transformations. Recent studies have shown that jointly learning both types of representations is often beneficial for downstream tasks, typically by employing separate projection heads. However, this design overlooks information shared between invariant and equivariant learning, which leads to redundant feature learning and inefficient use of model capacity. To address this, we introduce Soft Task-Aware Routing (STAR), a routing strategy for projection heads that models them as experts. STAR induces the experts to specialize in capturing either shared or task-specific information, thereby reducing redundant feature learning. We validate this effect by observing lower canonical correlations between invariant and equivariant embeddings. Experimental results show consistent improvements across diverse transfer learning tasks. The code is available at https://github.com/YonseiML/star.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。