提出新方法诊断世界模型对视觉扰动的鲁棒性。
Diagnosing JEPA World Models with Action-Conditioned Predictive Consistency
- 用动作条件预测一致性衡量扰动前后预测差异。
- 在4个视觉控制任务中验证预测误差与规划成本变化。
- 适合研究世界模型鲁棒性的研究人员使用。
联合嵌入预测架构(JEPAs)在紧凑潜在空间中学习世界模型,避免建模无关外观,但对视觉扰动仍无保障:扰动可改变编码表示并影响后续动作条件预测。双仿射性(Bisimulation)精确捕捉此要求:仅当动作条件后果一致时,两个观测才应视为同一状态。基于此,我们提出动作条件预测一致性(ACPC),通过测量干净历史与受视觉扰动的历史在相同动作序列下滚动前推后的偏离程度进行诊断。我们证明该偏离界定了扰动引起的多步预测误差和规划成本变化。在此基础上,定义两个互补度量:不变半径(IR)总结干净与扰动轨迹的展开范围,分离率(SR)检验不同状态在滚动后是否仍可区分。在四个视觉控制任务上的实验表明,成对ACPC能有效预测扰动导致的预测与成本变化。在LeWM上,IR-SR筛选可跨任务迁移,且在模糊与缩放下仍具信息量。PLDM在不同架构下也表现出相似诊断趋势。
原文摘要 · Abstract (English)
Joint-embedding predictive architectures (JEPAs) learn world models that predict in a compact latent space rather than in pixels, reducing the pressure to model nuisance appearance. Yet this provides no guarantee against visual perturbations: they can still alter the encoded representation and affect subsequent action-conditioned predictions. Bisimulation captures this requirement precisely: two observations should be treated as the same state only when their action-conditioned consequences agree. Guided by this criterion, we introduce Action-Conditioned Predictive Consistency (ACPC), a diagnostic that measures how far a clean history and a visually perturbed view of it diverge after being rolled forward under the same action sequence. We prove that this divergence bounds the perturbation-induced change in multi-step prediction error and planner cost. Building on pairwise ACPC, we define two complementary measures: the Invariance Radius (IR) summarizes clean-perturbed rollout spread, while the Separation Rate (SR) checks whether different states remain distinguishable after rollout. Experiments on four visual control tasks show that pairwise ACPC predicts perturbation-induced prediction and cost changes. On LeWM, the IR-SR screen transfers across tasks, and the joint diagnostic remains informative under blur and resize. PLDM exhibits similar diagnostic trends under a different architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。