Graph-JEPA看似表现良好，实则学不到有用信息，研究揭示并修复了这一隐性失效问题。
When Graph-JEPA Learns the Wrong Thing: Diagnosing and Repairing Category-Conditional Collapse
- 通过方差分配分析发现模型将主成分全集中在子图身份上，而非语义特征。
- 修复后模型恢复14.377比特信息，接近理论上限14.379比特，但无法支持科学推理。
- 提出可复现的诊断工具与目标质量门控，适合模型可靠性研究者参考。
联合嵌入预测架构在线性探测和有效秩指标上均表现良好，但我们发现其表征中实际无可用实例信息。在修复后，新问题出现：修复后的度量在无结构信息的目标上饱和。实验基于包含57,903篇文章的科学推理图数据集，每篇为一个子图。Graph-JEPA在掩码预测任务中达到0.871的线性探测准确率和18-47的有效秩，但检索任务仅恢复0.00比特（MRR 1.9e-4 vs 随机期望1.99e-4，p=0.98）。三个上界基准在相同数据池中分别恢复+14.28、+14.34、+14.22比特，排除了语料、掩码、数据池和度量方法的问题。根源在于方差分配：冻结输入时86.05%方差在子图身份，0.40%在语义方面；训练潜变量则相反（0.39%与99.61%）。该现象是目标函数全局最优解，初始即存在。修复配置达14.377/14.379比特，高于13.865比特的奥数基准。若将损失改为回归，性能降至0.307比特，验证机制正确。但修复后目标仍可约化（因子图内边由节点统计决定），且推理探针与比特数无相关性。数据目标质量不佳：25.96%节点为重复占位符，其余过于泛化。秩、探测和度量均可在非支持性评估上饱和。我们发布包含可约性审计和目标质量门控的测试框架。
原文摘要 · Abstract (English)
Joint-embedding predictive architectures are selected almost universally by linear probing and effective rank. We report a case where both read healthily while the representation carries zero usable instance information. We repair it, and a second failure appears: the repaired metric saturates on a target carrying no structural information. Our corpus is a scientific-reasoning graph over 57,903 articles, each a subgraph. A Graph-JEPA predicts one masked aspect from a subgraph's remaining aspects, attaining linear-probe accuracy 0.871 and effective rank 18-47, yet retrieval recovers 0.00 of 14.4 bits (MRR 1.9e-4 vs chance 1.99e-4, p=0.98). Three upper bounds on the same pool and code recover nearly everything (+14.28, +14.34, +14.22 bits), ruling out corpus, masking, pool, and metric as causes. We trace this to variance allocation - frozen inputs place 86.05% of variance on subgraph identity and 0.40% on aspect identity, while trained latents place 0.39% and 99.61%. This is a property of the objective's optimum: the degenerate solution is a global minimum of the coupled predictor/EMA-target objective, present already at init. A repaired configuration reaches 14.377 of 14.379 bits, above the 13.865-bit oracle; reverting the loss to regression drops it to 0.307 bits, confirming it. Yet the repair licenses nothing about reasoning: the target is reducible, since intra-subgraph edges are a deterministic function of node census. The oracle reaches 96.4% of the ceiling, and our largest effect is the learning-rate schedule, not architecture. Bits and a reasoning probe show no relation across ten cells. A data-derived target fails a quality gate - 25.96% of nodes are duplicate placeholders, and the rest is more generic than supporting evidence. Rank, probes, and metrics can all saturate on an unsupportive evaluation. We release a harness with a reducibility audit and target gate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。