提出新模型DeepDIVE,让多任务学习更高效且互不干扰。
Disentangled Interleaving Variational Encoding
- 将输入分解为边缘与条件概率分布,实现特征解耦
- 在两个公开数据集上预测精度优于原始VAE,接近顶尖水平
- 理论支撑强,适合研究多任务表征学习的学者
多任务学习中目标冲突带来挑战,需精心设计以避免任务间负面干扰。受概率论中边缘与条件分布启发,我们提出一种基于变分自编码器隐空间的解耦方法。所提模型DeepDIVE从原始输入中学习解耦特征,形成嵌入空间中的簇,并在融合阶段通过交叉注意力机制统一特征。理论上证明重建与预测目标联合可完整捕获下界,并基于朴素贝叶斯推导出解耦损失函数。假设先验为对数凹分布混合时,证明后验与先验间的KL散度由交叉熵最小化者上界控制,据此采用径向基函数与交叉熵结合的交错训练策略,为模型收敛提供理论依据。在两个公开数据集上的实验表明,DeepDIVE能有效解耦输入,预测精度优于原始VAE,接近现有最优基准。
原文摘要 · Abstract (English)
Conflicting objectives present a considerable challenge in interleaving multi-task learning, necessitating the need for meticulous design and balance to ensure effective learning of a representative latent data space across all tasks without mutual negative impact. Drawing inspiration from the concept of marginal and conditional probability distributions in probability theory, we design a principled and well-founded approach to disentangle the original input into marginal and conditional probability distributions in the latent space of a variational autoencoder. Our proposed model, Deep Disentangled Interleaving Variational Encoding (DeepDIVE) learns disentangled features from the original input to form clusters in the embedding space and unifies these features via the cross-attention mechanism in the fusion stage. We theoretically prove that combining the objectives for reconstruction and forecasting fully captures the lower bound and mathematically derive a loss function for disentanglement using Naïve Bayes. Under the assumption that the prior is a mixture of log-concave distributions, we also establish that the Kullback-Leibler divergence between the prior and the posterior is upper bounded by a function minimized by the minimizer of the cross entropy loss, informing our adoption of radial basis functions (RBF) and cross entropy with interleaving training for DeepDIVE to provide a justified basis for convergence. Experiments on two public datasets show that DeepDIVE disentangles the original input and yields forecast accuracies better than the original VAE and comparable to existing state-of-the-art baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。