让自监督学习的表示自动匹配指定分布，提升生成与判别能力。
Efficient Distribution Matching of Representations via Noise-Injected Deep InfoMax
- 在编码器输出加独立噪声，保持原有信息最大化目标
- 可学习均匀、正态及其他连续分布的表示，适配多种下游任务
- 兼顾下游性能与分布匹配质量，适合生成与异常检测场景
深度信息最大化（Deep InfoMax, DIM）是一种基于输入与编码器输出间互信息最大化的自监督表示学习方法。尽管对比学习与DIM已被广泛研究，但如何使学习到的表示符合特定先验分布（即分布匹配，DM）仍缺乏有效方法。鉴于分布匹配对生成建模、解耦表示、异常检测等任务的重要性，本文提出在编码器归一化输出中注入独立噪声，同时保留原有的信息最大化训练目标，从而实现对选定先验分布的自动匹配。实验表明，该方法可学习均匀分布、正态分布及其他绝对连续分布的表示。在多个下游任务上的测试结果表明，模型在下游任务表现与分布匹配质量之间存在适度权衡。
原文摘要 · Abstract (English)
Deep InfoMax (DIM) is a well-established method for self-supervised representation learning (SSRL) based on maximization of the mutual information between the input and the output of a deep neural network encoder. Despite the DIM and contrastive SSRL in general being well-explored, the task of learning representations conforming to a specific distribution (i.e., distribution matching, DM) is still under-addressed. Motivated by the importance of DM to several downstream tasks (including generative modeling, disentanglement, outliers detection and other), we enhance DIM to enable automatic matching of learned representations to a selected prior distribution. To achieve this, we propose injecting an independent noise into the normalized outputs of the encoder, while keeping the same InfoMax training objective. We show that such modification allows for learning uniformly and normally distributed representations, as well as representations of other absolutely continuous distributions. Our approach is tested on various downstream tasks. The results indicate a moderate trade-off between the performance on the downstream tasks and quality of DM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。