让生成图像保持可检索,解决多尺度下语义丢失问题。
When Images Look Right and Retrieve Wrong: Coverage-Guided Cross-Scale Re-Indexing for Knowledge-Faithful Generative Perception

- 构建三层语义金字塔,按尺度路由注意力以保留细节
- 在极端尺度变化场景中提升检测精度达9.7 mAP
- 适合需要精准图文检索的生成系统开发者
多模态系统将生成图像重新通过原始视觉-语言索引检索,要求输出内容仍能被原查询召回。当场景包含极不同时,现有语言引导生成器依赖全局文本嵌入,忽略尺度相关概念,导致即使像素质量高也无法正确检索。本文提出CERES框架,通过三层次语义金字塔、共现感知路由器、尺度定向交叉注意力和轻量U-Net生成器,实现闭环索引。使用可微分soft-Jaccard覆盖目标(0.39M参数生成器),并通过DINOv2线性探测器验证覆盖。在四个全色融合基准的七种设置中,取得新最优表现,极端尺度下检测指标提升最高达+4.64%相对Q2n和+9.7 mAP(DOTA)。概念查询召回率提升14.0点,图文平均倒数排名提高0.19,证明闭环设计有效保留了可检索语义而非仅特征一致性。
原文摘要 · Abstract (English)
Multimodal information systems increasingly route generated visual content back through the same vision-language index that informed its production, so the output must remain retrievable by the queries it was meant to serve. When the scene contains entities at vastly different scales, existing language-guided generators condition on a single, globally pooled text embedding and quietly drop scale-specific concepts, breaking concept-query retrieval even when pixel fidelity is high. We formalise this failure as semantic collapse and propose CERES, a closed-loop multimodal indexing framework that builds a three-level semantic pyramid, mines implicit concepts via a co-occurrence-aware router, performs scale-routed cross-attention into a lightweight U-Net generator, and verifies coverage by re-indexing the generated image with the same frozen VLM. A continuously differentiable soft-Jaccard coverage objective returns dense gradients to the 0.39 M-parameter generator under explicit non-degeneracy conditions, and coverage is verified by an independent DINOv2 linear probe trained only on external scene and object labels. On four pansharpening benchmarks across seven settings, CERES delivers the new state of the art with the largest gains where scale variation is most extreme (+4.64% relative Q2n and +9.7 mAP for DOTA detection). It also improves concept-query retrieval Recall@5 by +14.0 points and image-text mean reciprocal rank by 0.19 over the strongest baseline, showing that the closed loop preserves queryable content rather than self-referential feature consistency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。