arXiv:2608.03025cs.AI2026-08

用扩散模型生成视觉特征,更准确验证图文实体类型。

DiffImaginE: Imagine to Verify Entity Types with Diffusion

论文配图:DiffImaginE: Imagine to Verify Entity Types with Diffusion
图 1 · 摘自论文原文
  • 将实体类型验证转为条件扩散推理,通过去噪误差评分
  • 在Twitter-2015/2017上比确定性方法提升性能,显著优于基线
  • 适合做图文实体识别、需要概率解释的场景

多模态命名实体识别(MNER)判断每个候选文本片段及其类型假设是否由文本和视觉证据共同支持。现有‘想象-对比’验证器将每个(片段,类型)对映射为单一预测视觉特征,将多样的视觉表现压缩成单一原型,且不提供显式的概率语义。我们提出DiffImaginE,将MNER类型验证建模为条件潜空间扩散推断。给定局部化视觉证据,类型条件去噪器预测其标准潜变量中注入的噪声。由此产生的去噪误差提供了一个与ELBO一致的类型条件负对数似然替代量,使不同类型假设可通过其对观测的解释能力进行排序。DiffImaginE保留标准多模态编码器堆栈,将确定性验证器替换为使用最小信噪比(Min-SNR)加权训练的无分类器引导扩散评分器。直接监督每类扩散得分作为分类逻辑值,学习跨噪声层级的聚合,并采用反向采样以降低蒙特卡洛比较方差。分析表明,无分类器引导能锐化诱导的类型后验,且刻画了反向配对在相同去噪器成本下降低方差的条件。在Twitter-2015和Twitter-2017上的实验显示,相较于相同编码器、辅助目标和评估协议下的确定性ImaginE对照组,性能持续提升,且经消融实验与配对显著性检验验证。

原文摘要 · Abstract (English)

Multimodal named entity recognition (MNER) determines whether each candidate span and entity-type hypothesis is supported by joint textual and visual evidence. Existing imagine-and-compare verifiers map each (span, type) pair to one predicted visual feature, compressing diverse visual realisations into a single prototype and providing a compatibility score without explicit probabilistic semantics. We introduce DiffImaginE, which formulates MNER type verification as conditional latent diffusion inference. Given span-localised visual evidence, a type-conditioned denoiser predicts noise injected into its standardised latent. The resulting denoising error provides an ELBO-consistent surrogate for type-conditional negative log-likelihood, allowing competing type hypotheses to be ranked by how well they explain the observation. DiffImaginE retains a standard multimodal encoder stack and replaces the deterministic verifier with a classifier-free-guided diffusion scorer trained using Min-SNR weighting. We directly supervise per-type diffusion scores as classification logits, learn aggregation across noise levels, and use antithetic sampling to reduce Monte Carlo comparison variance. Our analysis shows that classifier-free guidance sharpens the induced type posterior and characterises when antithetic pairing reduces variance at equal denoiser cost. Experiments on Twitter-2015 and Twitter-2017 show consistent gains over a matched deterministic ImaginE control under the same encoder, auxiliary objectives, and evaluation protocol, supported by ablations and paired significance tests.

多模态扩散模型实体识别

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。