arXiv:2605.00960cs.CVcs.CL2026-05

用能量模型检测跨模态结构异常,无需重新训练编码器

Energy-Based Constraint Networks: Learning Structural Coherence Across Modalities

  • 通过双头注意力状态空间模型计算结构一致性能量
  • 文本任务93.4%准确率,视觉检测AUC达0.959
  • 可灵活组合不同故障类型检测分支,适配多场景

我们提出基于能量的约束网络——一种与模态无关的架构,能从对比样本中学习结构一致性。系统处理冻结编码器嵌入,通过带有双头注意力的状态空间模型,输出衡量结构一致性的标量能量及每位置的能量得分,用于定位异常。多个独立训练的分支可检测不同类型异常,并在推理时无干扰组合。在文本领域,使用冻结BERT和740万可训练参数,对已知污染类型达到93.4%准确率,对9种未见过的类型达87.2%。在视觉领域,采用冻结DINOv2和每分支360万参数,在未使用Celeb-DF数据的情况下,于FaceForensics++ Deepfakes上实现0.959 AUC,Celeb-DF上达0.870。该框架支持灵活训练:分支可基于设计者指定的污染、真实配对数据或两者共同学习。可组合分支要求表示兼容性——实验验证五种不兼容方法失败后,兼容方案成功。该架构编码器无关且领域无关:换领域只需新污染策略,换编码器仅需新输入投影层。据我们所知,这是首个将模内结构一致性显式建模为能量景观并实现逐位置分解的架构,也是首个仅通过重定义污染策略即实现跨模态迁移的架构。

原文摘要 · Abstract (English)

We introduce energy-based constraint networks -- a modality-agnostic architecture that learns structural coherence from contrastive pairs. The system processes frozen encoder embeddings through a state-space model with dual-head attention, producing a scalar energy measuring structural consistency alongside per-position energy scores that localize violations. Multiple independently trained branches detect different violation types and compose at inference without interference. We demonstrate the framework in two domains. In text, the system achieves 93.4% accuracy on trained corruption types and 87.2% on 9 unseen types, using frozen BERT and 7.4M trainable parameters. In vision, the same architecture achieves competitive deepfake detection: 0.959 AUC on FaceForensics++ Deepfakes and 0.870 on Celeb-DF without any Celeb-DF training data, using frozen DINOv2 and 3.6M parameters per branch. The framework supports flexible training: branches learn from designer-specified corruptions, real-world paired data, or both. Composable branches require representation compatibility -- a finding validated through extensive experimentation where five incompatible approaches failed before the compatible one succeeded. The architecture is encoder-agnostic and domain-agnostic: changing the domain requires only new corruption strategies; changing the encoder requires only a new input projection layer. To our knowledge, this is the first architecture to learn within-modality structural coherence as an explicit energy landscape with per-position decomposition, and to demonstrate that the same architecture transfers across modalities via corruption respecification alone.

能量模型结构检测跨模态异常识别

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。