提出近似可识别性理论,解释表示学习中表征为何稳定且可解耦。
Statistical and structural identifiability in representation learning
- 区分统计与结构近似可识别性,放宽对精确识别的苛求
- 证明非线性解码器模型中间表征具有ε-近似可识别性
- 用ICA后处理实现最优解耦,适用于自编码器和基础模型
表示学习模型展现出惊人的内部表征稳定性。以往研究常将其视为单一属性,本文将其形式化为两个独立概念:统计可识别性(跨运行的一致性)与结构可识别性(与未观测真实结构的对齐)。鉴于现代表示学习模型难以实现完全逐点可识别,我们提出新的、不依赖模型的统计与结构近似可识别性定义,允许误差容忍度ε。基于此,我们证明了具非线性解码器的模型其表征具有统计ε-近似可识别性,将现有可识别性理论从生成式预训练变压器(GPTs)的最后一层扩展至包括(掩码)自编码器(MAEs)和监督学习者在内的广泛模型的中间表示。尽管弱化假设带来较弱可识别性,我们证明独立成分分析(ICA)可解决大部分剩余线性模糊性,并通过实证验证了近似可识别性。在额外数据生成过程假设下,统计可识别性可延伸至结构可识别性,从而给出一种简单实用的解耦方案:对潜在表征进行ICA后处理。在合成基准上,该方法使用普通自编码器即达前沿解耦性能;在细胞显微镜的基础模型规模MAE上,成功分离生物变异与技术批次效应,显著提升下游泛化能力。
原文摘要 · Abstract (English)
Representation learning models exhibit a surprising stability in their internal representations. Whereas most prior work treats this stability as a single property, we formalize it as two distinct concepts: statistical identifiability (consistency of representations across runs) and structural identifiability (alignment of representations with some unobserved ground truth). Recognizing that perfect pointwise identifiability is generally unrealistic for modern representation learning models, we propose new model-agnostic definitions of statistical and structural near-identifiability of representations up to some error tolerance $ε$. Leveraging these definitions, we prove a statistical $ε$-near-identifiability result for the representations of models with nonlinear decoders, generalizing existing identifiability theory beyond last-layer representations in e.g. generative pre-trained transformers (GPTs) to near-identifiability of the intermediate representations of a broad class of models including (masked) autoencoders (MAEs) and supervised learners. Although these weaker assumptions confer weaker identifiability, we show that independent components analysis (ICA) can resolve much of the remaining linear ambiguity for this class of models, and validate and measure our near-identifiability claims empirically. With additional assumptions on the data-generating process, statistical identifiability extends to structural identifiability, yielding a simple and practical recipe for disentanglement: ICA post-processing of latent representations. On synthetic benchmarks, this approach achieves state-of-the-art disentanglement using a vanilla autoencoder. With a foundation model-scale MAE for cell microscopy, it disentangles biological variation from technical batch effects, substantially improving downstream generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。