用GAN生成文本并半监督微调Transformer，提升生成质量。
A Semi-Supervised Text Generation Framework Combining a Deep Transformer and a GAN
- 将24层Transformer与GAN结合，先预训练再用生成数据微调。
- 通过Gumbel-Softmax处理离散词元，实现梯度传递与生成优化。
- 适合需要少量标注数据却想提升生成效果的研究者。
本文提出一种结合深度生成预训练Transformer与生成对抗网络（GAN）的半监督文本生成框架。模型首先在大规模多样化语料上进行无监督预训练，使用24层Transformer结构。随后引入一个简单的GAN架构用于合成文本生成，并采用Gumbel-Softmax方法处理词元的离散性问题。论文还提出一种半监督策略：将真实数据与GAN生成样本合并，用于微调Transformer模型。文中包含对最小最大目标函数的详细理论推导，以及对Gumbel-Softmax重参数化技巧的深入讨论。
原文摘要 · Abstract (English)
This paper introduces a framework that connects a deep generative pre-trained Transformer language model with a generative adversarial network for semi-supervised text generation. In other words, the proposed model is first pre-trained unsupervised on a large and diverse text corpus with 24 layers. Then a simple GAN architecture for synthetic text generation is introduced, and Gumbel-Softmax is applied to handle the discreteness of tokens. The paper also shows a semi-supervised approach where real data is augmented with GAN samples, which is further used to fine-tune the Transformer model on the merged dataset. Detailed theoretical derivations are also included, outlining the proof of the min-max objective function, and an extensive discussion of the Gumbel-Softmax reparameterization trick.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。