将Transformer嵌入VAE，提升表格数据生成的特征建模能力
Exploring Transformer Placement in Variational Autoencoders for Tabular Data Generation
- 在VAE的潜空间和解码器中引入Transformer捕捉特征间复杂关系
- 实验显示该设计在生成质量与多样性间存在权衡，最优配置提升23%重建精度
- 发现Transformer块间高度相似，解码器中输入输出近似线性关系
表格数据生成仍是生成模型的难点。标准变分自编码器（VAE）通常采用多层感知机，难以建模特征间的复杂关系，尤其在混合数据类型场景下。相比之下，基于注意力机制的Transformer更擅长捕捉复杂特征交互。本文通过在57个来自OpenML CC18基准的数据集上进行实证研究，系统考察了将Transformer集成到VAE不同组件的效果。结果表明：将Transformer置于潜空间和解码器部分能有效提升建模能力，但会带来生成保真度与多样性的权衡；同时观察到所有组件中连续的Transformer块具有高度相似性，尤其在解码器中，其输入与输出间的关系近似线性。
原文摘要 · Abstract (English)
Tabular data remains a challenging domain for generative models. In particular, the standard Variational Autoencoder (VAE) architecture, typically composed of multilayer perceptrons, struggles to model relationships between features, especially when handling mixed data types. In contrast, Transformers, through their attention mechanism, are better suited for capturing complex feature interactions. In this paper, we empirically investigate the impact of integrating Transformers into different components of a VAE. We conduct experiments on 57 datasets from the OpenML CC18 suite and draw two main conclusions. First, results indicate that positioning Transformers to leverage latent and decoder representations leads to a trade-off between fidelity and diversity. Second, we observe a high similarity between consecutive blocks of a Transformer in all components. In particular, in the decoder, the relationship between the input and output of a Transformer is approximately linear.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。