提出基于哈密顿几何的JEPA,让模型更精准捕捉视觉特征间动态关系。
Beyond Isotropy in JEPAs: Hamiltonian Geometry and Symplectic Prediction

- 用相空间和哈密顿跃迁映射替代传统各向同性编码,建模视图间动态演化
- 在CIFAR-100上比SIGReg提升6.45点kNN@20,ImageNet-100提升7.52点线性探测
- 适合关注表征几何结构与自监督学习中耦合机制的研究者
JEPAs通常将单视角嵌入正则化为各向同性高斯分布,隐式引入欧氏对称性。我们证明这并非无害默认:当下游几何结构已知且正定($H\succ0$)时,在哈密顿能量预算下,最小最大熵协方差为$(c/d)H^{-1}$,欧氏各向同性会带来可计算的偏差代价。更重要的是,当下游几何未知时,任何固定的边际目标均非最优——每种固定协方差形状都可能与某种结构化几何完全错位。进一步发现,即使使用理想的单视角边际,也无法确定视图间预测耦合关系。因此建议将结构偏差引入跨视图耦合而非固定编码器边际。我们提出 extbf{HamJEPA},将每个视图编码为相空间状态$(q,p)$,通过学习的哈密顿跃迁步映射预测视图间转移,同时非各向同性尺度与谱底限防止崩溃。在去头令牌协议下,HamJEPA在30轮时较SIGReg提升+4.89 kNN@20和+3.52线性探测点,80轮时提升+6.45 kNN@20和+10.64线性探测点;匹配的MLP预测器消融实验表明,辛耦合是邻域几何增益的关键。在ImageNet-100上,HamJEPA-$q$在45轮时提升+4.82 kNN@20和+7.52线性探测点。
原文摘要 · Abstract (English)
JEPAs often regularize one-view embeddings toward an isotropic Gaussian, implicitly baking Euclidean symmetry into the representation. We show that this is not merely a benign default. For a known structured downstream geometry $H\succ0$, the minimax and maximum-entropy covariance under a Hamiltonian energy budget is $(c/d)H^{-1}$, and Euclidean isotropy incurs a closed-form price of isotropy. More importantly, when the downstream geometry is unknown, no geometry-independent fixed marginal target is canonical: every fixed covariance shape can be maximally misaligned for some structured geometry. We further show that even oracle one-view marginals do not identify the JEPA view-to-view predictive coupling. These results suggest that the structural bias in JEPAs should enter the cross-view coupling rather than a fixed encoder marginal. We instantiate this principle with \textbf{HamJEPA}, which encodes each view as a phase-space state $(q,p)$ and predicts view-to-view transitions with a learned Hamiltonian leapfrog map, while non-isotropic scale and spectral floors prevent collapse. In a deliberately headless token protocol, HamJEPA improves over SIGReg on CIFAR-100 by $+4.89$ kNN@20 and $+3.52$ linear-probe points at 30 epochs, and by $+6.45$ kNN@20 and $+10.64$ linear-probe points at 80 epochs, while a matched MLP predictor ablation shows that the symplectic coupling is the ingredient driving the neighborhood-geometry gain. On ImageNet-100, HamJEPA-$q$ improves by $+4.82$ kNN@20 and $+7.52$ linear-probe points at 45 epochs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。