提出新框架解决弱监督指代表达理解中多目标与零目标问题。
LIHE: Linguistic Instance-Split Hyperbolic-Euclidean Framework for Generalized Weakly-Supervised Referring Expression Comprehension
- 分两阶段处理:先预测目标数并拆解表达式,再用混合几何定位
- 在gRefCOCO和Ref-ZOM上建立首个有效的弱监督泛化基准
- 融合双曲与欧氏几何,避免语义坍塌,提升细粒度区分能力
现有弱监督指代表达理解(WREC)方法受限于一一对应假设,难以处理真实场景中零个或多个目标的情况。为此,我们提出更实用的广义弱监督指代表达理解任务(WGREC),支持可变数量的目标。但扩展至WGREC面临两大挑战:监督信号模糊(图像级弱监督不足以推断目标数量与身份)和语义表征坍塌(标准欧氏相似性使层次相关概念聚成无区分簇)。为此,我们提出新型框架LIHE,分两阶段:第一阶段“指代解耦”预测目标数并分解复杂表达;第二阶段“指代定位”使用创新的HEMix混合相似性模块,结合欧氏精确对齐与双曲几何层次建模优势,有效防止语义坍塌并保留细粒度区分。大量实验表明,LIHE在gRefCOCO和Ref-ZOM上建立了首个有效的弱监督WGREC基线,HEMix在标准REC基准上平均提升[email protected]达2.5%。代码已开源。
原文摘要 · Abstract (English)
Existing Weakly-Supervised Referring Expression Comprehension (WREC) methods, while effective, are fundamentally limited by a one-to-one mapping assumption, hindering their ability to handle expressions corresponding to zero or multiple targets in realistic scenarios. To bridge this gap, we introduce the Weakly-Supervised Generalized Referring Expression Comprehension task (WGREC), a more practical paradigm that handles expressions with variable numbers of referents. However, extending WREC to WGREC presents two fundamental challenges: supervisory signal ambiguity, where weak image-level supervision is insufficient for training a model to infer the correct number and identity of referents, and semantic representation collapse, where standard Euclidean similarity forces hierarchically-related concepts into non-discriminative clusters, blurring categorical boundaries. To tackle these challenges, we propose a novel WGREC framework named Linguistic Instance-Split Hyperbolic-Euclidean (LIHE), which operates in two stages. The first stage, Referential Decoupling, predicts the number of target objects and decomposes the complex expression into simpler sub-expressions. The second stage, Referent Grounding, then localizes these sub-expressions using HEMix, our innovative hybrid similarity module that synergistically combines the precise alignment capabilities of Euclidean proximity with the hierarchical modeling strengths of hyperbolic geometry. This hybrid approach effectively prevents semantic collapse while preserving fine-grained distinctions between related concepts. Extensive experiments demonstrate LIHE establishes the first effective weakly supervised WGREC baseline on gRefCOCO and Ref-ZOM, while HEMix achieves consistent improvements on standard REC benchmarks, improving [email protected] by up to 2.5\%. The code is available at https://anonymous.4open.science/r/LIHE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。