测试统一模型在图文交替生成中的语义漂移问题
The Telephone Game: Evaluating Semantic Drift in Unified Models
- 通过模拟电话游戏设计多轮图文转换评估协议
- 发现模型在多轮交互中语义丢失严重，关键信息常被忽略
- 适合关注统一模型可靠性与实际应用的开发者
统一模型（UMs）将视觉理解（I2T）与生成（T2I）集成于单一框架。本文聚焦T2I与I2T任务，强调跨一致性——模型理解的内容应能准确生成。现有基准分别评估两项能力：用FID/GenEval测T2I；用MME/MMBench测I2T。我们发现此分离评估存在重大缺陷：在单项指标上表现优异的模型，在组合使用时会严重丢失实体、属性、空间关系和数量，导致语义漂移。为此提出语义漂移协议（SDP），借鉴电话游戏思想，通过多轮交替I2T与T2I，量化语义保持程度。引入均累积漂移（MCD）衡量三类表示空间中的内容保留率，并扩展GenEval为多轮生成评估（MGG）。构建包含400组图像-文本对的新基准，数据源自NoCaps与DOCCI，侧重新物体与细粒度描述。对七种模型的应用显示，漂移差异显著，且无法由单轮得分预测：BAGEL在多轮中保持高保真度，而VILA-U与Janus变体在五轮内即崩溃，尽管其孤立指标相近。识别出六类典型失败模式，且退化多为灾难性而非渐进式——一旦出现关键错误，后续生成将不断放大。SDP揭示了单轮基准所忽视的故障模式，可更真实评估统一模型的可靠性。代码与数据集见：https://github.com/mollahsabbir/telephone-game-semantic-drift
原文摘要 · Abstract (English)
Unified models (UMs) combine visual understanding (I2T) and generation (T2I) in a single framework. We focus on T2I and I2T, where cross-consistency---what a model understands, it should be able to generate---is a promise of unification and a necessity when composing both capabilities. Yet, existing benchmarks evaluate them in isolation: FID/GenEval for T2I; MME/MMBench for I2T. We show this gap is consequential: models scoring competitively on these benchmarks can fail severely when understanding and generation are composed, losing entities, attributes, spatial relations, and counts, resulting in semantic drift. To quantify drift, we introduce the Semantic Drift Protocol (SDP), inspired by the Telephone Game: starting from a caption or image, we alternate I2T and T2I over multiple generations and measure semantic preservation. We propose Mean Cumulative Drift (MCD), an embedding-based measure of content retention across three representation spaces, and Multi-Generation GenEval (MGG), extending GenEval's object-level compliance scoring across generations. To stress-test models beyond COCO-style data, we create a benchmark of 400 image-text pairs sampled from NoCaps and DOCCI, emphasizing novel objects and fine-grained descriptions. Applying SDP to seven models reveals that drift varies dramatically and is not predicted by single-pass scores: BAGEL retains high semantic fidelity over multiple generations, while VILA-U and Janus variants collapse within five generations, despite comparable isolated metrics. We identify six recurring failure modes and find degradation is typically catastrophic rather than gradual: once a critical error occurs, subsequent generations compound it. SDP exposes failure modes that single-pass benchmarks miss, enabling a more faithful assessment of unified model reliability. Code and benchmark: https://github.com/mollahsabbir/telephone-game-semantic-drift
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。