用信息瓶颈理论解析生成模型的多样性差异。
Deconstructing Generative Diversity: An Information Bottleneck Analysis of Discrete Latent Generative Models
- 从压缩与多样性的冲突出发，拆解生成策略
- 发现三类模型各有侧重：优先多样、优先压缩、解耦生成
- 提出无训练干预法，可直接增强生成多样性
离散潜在生成模型（如AR、MIM、Diffusion）在生成多样性上表现差异显著。本文基于信息瓶颈（IB）理论构建诊断框架，将生成过程建模为‘压缩压力’（降低码本熵）与‘多样性压力’（给定输入时最大化条件熵）之间的权衡。进一步将多样性分解为‘路径多样性’（高层生成策略选择）和‘执行多样性’（选定策略后的随机执行）。为此提出三种零样本、推理时干预方法，直接扰动潜在生成过程以揭示模型如何分配与表达多样性。对代表性AR、MIM和Diffusion系统的应用揭示三类不同策略：‘多样性优先’（MIM）、‘压缩优先’（AR）、‘解耦’（Diffusion）。该分析为模型行为差异提供理论解释，并启发一种新型推理时多样性增强技术。
原文摘要 · Abstract (English)
Generative diversity varies significantly across discrete latent generative models such as AR, MIM, and Diffusion. We propose a diagnostic framework, grounded in Information Bottleneck (IB) theory, to analyze the underlying strategies resolving this behavior. The framework models generation as a conflict between a 'Compression Pressure' - a drive to minimize overall codebook entropy - and a 'Diversity Pressure' - a drive to maximize conditional entropy given an input. We further decompose this diversity into two primary sources: 'Path Diversity', representing the choice of high-level generative strategies, and 'Execution Diversity', the randomness in executing a chosen strategy. To make this decomposition operational, we introduce three zero-shot, inference-time interventions that directly perturb the latent generative process and reveal how models allocate and express diversity. Application of this probe-based framework to representative AR, MIM, and Diffusion systems reveals three distinct strategies: "Diversity-Prioritized" (MIM), "Compression-Prioritized" (AR), and "Decoupled" (Diffusion). Our analysis provides a principled explanation for their behavioral differences and informs a novel inference-time diversity enhancement technique.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。