arXiv:2607.24176cs.CLcs.LG2026-07

通过分阶段诊断发现,压缩文本生成中质量损失主要源于编码器而非生成器。

Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization

  • 构建分阶段验证流程,分离编码器重建与潜在空间生成的性能
  • 编码器失真使困惑度中位数上升80.4%,显示问题在生成前已出现
  • 适合关注生成系统瓶颈定位的研究者与工程优化人员

压缩短文本生成器可能在两个环节失败:编码器在生成前丢弃信息,或潜在生成器产生弱编码。若不区分这两种失效模式,研究者可能浪费算力优化错误组件。本文在64到16的TinyStories可控案例中研究该问题,采用分层VQ-VAE-2编码器与掩码离散扩散生成器(MDLM)。通过分阶段验证协议,在统一外部GPT-2评分器下分别评估编码器重建保真度、潜在生成质量及辅助潜变量诊断,并报告互补语义指标用于几何分析。测试中,仅编码器重建使中位数困惑度从15.17升至27.36(+80.4%),p95从25.10升至98.91(+294.1%),表明主要质量损失发生在潜在生成之前。同一评分器下,代码空间MDLM相比词元空间扩散,平均、中位和p95困惑度分别降低32.9%、30.9%、36.6%。几何感知正则化虽改善局部潜变量代理指标,但未提升解码文本指标。贡献为方法论而非算法:提出可复用的分阶段诊断框架,揭示在此设定下,编码器保真度而非潜在去噪决定实际质量上限。

原文摘要 · Abstract (English)

Compressed short-text generators can fail in two different places: the codec may discard information before generation starts, or the latent generator may produce weak codes. Without separating these failure modes, researchers can spend compute improving the wrong component. We study this problem in a controlled 64-to-16 TinyStories case study built from a hierarchical VQ-VAE-2 codec and a masked discrete diffusion generator (MDLM). We use a staged validation protocol that separates codec reconstruction fidelity, latent generation quality, and auxiliary latent diagnostics under one shared external GPT-2 scorer, while reporting complementary semantic metrics for the geometry study. In the tested configuration, codec reconstruction alone raises median external perplexity from 15.17 to 27.36 (+80.4%) and p95 from 25.10 to 98.91 (+294.1%), showing that the dominant quality loss appears before latent generation begins. Under the same scorer, code-space MDLM remains materially stronger than token-space diffusion, reducing mean, median, and p95 by 32.9%, 30.9%, and 36.6%, respectively. Geometry-aware regularization improves local latent proxies but does not improve decoded-text metrics in the available runs. The contribution is methodological rather than algorithmic: the paper presents a reusable staged diagnosis for one concrete pipeline and shows that, in this setting, codec fidelity rather than latent denoising sets the practical quality ceiling.

文本生成模型诊断压缩生成瓶颈定位

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。