arXiv:2512.19693cs.CV2025-12被引 18

提出统一自编码框架,让语义与像素信息在频域协同共存。

The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding

  • 通过频带调制器在统一潜空间中融合语义与细节
  • 在单个潜空间实现语义抽象与像素保真双提升
  • 适用于多模态生成,尤其适合需要高保真输出的场景

跨模态深度表征具有内在关联性。本文系统分析了多种语义与像素编码器的频谱特性,发现其特征频谱与其功能角色存在显著对应关系:语义编码器主要捕捉低频成分以表达抽象意义,而像素编码器则额外保留高频信息以传递细粒度细节。这一启发性发现揭示了编码器行为与频谱结构之间的深层联系,我们将其定义为‘棱镜假说’——各数据模态可视为自然世界在共享特征频谱上的投影,如同棱镜分光。基于此,我们提出统一自编码(UAE),通过创新的频带调制机制,实现语义结构与像素细节的和谐共存。大量实验表明,UAE 能有效在单一潜空间中统一语义抽象与像素级保真,达到当前最优性能;同时,该方法可直接应用于像素空间建模,显著优于基线模型 JIT,FID 和 IS 指标均有明显提升。

原文摘要 · Abstract (English)

Deep representations across modalities are inherently intertwined. In this paper, we systematically analyze the spectral characteristics of various semantic and pixel encoders. Interestingly, our study uncovers a highly inspiring and rarely explored correspondence between an encoder's feature spectrum and its functional role: semantic encoders primarily capture low-frequency components that encode abstract meaning, whereas pixel encoders additionally retain high-frequency information that conveys fine-grained detail. This heuristic finding offers a unifying perspective that ties encoder behavior to its underlying spectral structure. We define it as the Prism Hypothesis, where each data modality can be viewed as a projection of the natural world onto a shared feature spectrum, just like the prism. Building on this insight, we propose Unified Autoencoding (UAE), a model that harmonizes semantic structure and pixel details via an innovative frequency-band modulator, enabling their seamless coexistence. Extensive experiments demonstrate that UAE effectively unifies semantic abstraction and pixel-level fidelity within a single latent space, achieving state-of-the-art performance. Moreover, we show that UAE can be directly applied to pixel-space modeling, significantly improving both FID and IS over the vanilla JIT baseline. Our code is avaliable at: https://github.com/WeichenFan/UAE.

多模态自编码频域建模统一表征

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。