arXiv:2602.03906cs.LGcs.AI2026-02

提出几何感知的信息瓶颈方法,直接控制信息压缩,提升模型稳定性和性能。

GeoIB: Geometry-Aware Information Bottleneck via Statistical-Manifold Compression

  • 基于信息几何,用精确投影形式替代传统互信息估计
  • 在多个数据集上实现更优的预测准确率与压缩比平衡
  • 适合关注模型鲁棒性与信息压缩机制的研究者

信息瓶颈(IB)广泛应用于深度学习,但通常依赖可计算的近似方法(如变分界或神经互信息估计器),而非直接控制互信息 I(X;Z)。此类方法存在松散和估计偏差问题,导致压缩效果间接且优化不稳定。本文从信息几何视角重审IB问题,提出几何感知信息瓶颈(GeoIB),无需互信息估计。我们证明 I(X;Z) 与 I(Z;Y) 可表示为联合分布到独立流形的最小KL距离的精确投影形式。基于此,GeoIB通过两个互补项控制信息压缩:(i) 分布层面的Fisher-Rao(FR)差异,二阶匹配KL并保持重参数化不变性;(ii) 几何层面的Jacobian-Frobenius(JF)项,通过惩罚编码器拉回体积膨胀,提供 I(Z;X) 的局部容量型上界。进一步推导出与FR度量一致的自然梯度优化器,并证明标准加法自然梯度步长一阶等价于测地线更新。大量实验表明,GeoIB在主流基准数据集上优于现有IB方法,在信息平面中实现了更优的预测准确率与压缩比权衡。该方法通过单一瓶颈乘子统一分布与几何正则化,提升了不变性与优化稳定性。源码已公开于 https://anonymous.4open.science/r/G-IB-0569。

原文摘要 · Abstract (English)

Information Bottleneck (IB) is widely used, but in deep learning, it is usually implemented through tractable surrogates, such as variational bounds or neural mutual information (MI) estimators, rather than directly controlling the MI I(X;Z) itself. The looseness and estimator-dependent bias can make IB "compression" only indirectly controlled and optimization fragile. We revisit the IB problem through the lens of information geometry and propose a \textbf{Geo}metric \textbf{I}nformation \textbf{B}ottleneck (\textbf{GeoIB}) that dispenses with mutual information (MI) estimation. We show that I(X;Z) and I(Z;Y) admit exact projection forms as minimal Kullback-Leibler (KL) distances from the joint distributions to their respective independence manifolds. Guided by this view, GeoIB controls information compression with two complementary terms: (i) a distribution-level Fisher-Rao (FR) discrepancy, which matches KL to second order and is reparameterization-invariant; and (ii) a geometry-level Jacobian-Frobenius (JF) term that provides a local capacity-type upper bound on I(Z;X) by penalizing pullback volume expansion of the encoder. We further derive a natural-gradient optimizer consistent with the FR metric and prove that the standard additive natural-gradient step is first-order equivalent to the geodesic update. We conducted extensive experiments and observed that the GeoIB achieves a better trade-off between prediction accuracy and compression ratio in the information plane than the mainstream IB baselines on popular datasets. GeoIB improves invariance and optimization stability by unifying distributional and geometric regularization under a single bottleneck multiplier. The source code of GeoIB is released at "https://anonymous.4open.science/r/G-IB-0569".

信息瓶颈信息几何模型压缩优化稳定

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。