用文本引导提取关键视觉特征，提升电商推荐的鲁棒性。
Text-Guided Visual Representation Learning for Robust Multimodal E-Commerce Recommendation
- 通过文本元数据指导视觉特征选择，分离有用与噪声信息。
- 在真实电商数据上，平均提升命中率6.04%（H@100）。
- 适合需要抗干扰视觉推荐的工业级应用。
多模态商品嵌入对电商商品-商品检索至关重要，但实际商品图像常含促销贴纸和背景杂乱，引入虚假视觉线索，降低检索鲁棒性。该问题在MLRM类流程中尤为突出：冻结的视觉编码器通过轻量连接器与大语言模型相连，连接器需有选择地聚合视觉标记。本文提出文本引导的Q-Former（TGQ-Former），利用结构化元数据作为语义引导，精准提取视觉标记并保留互补视觉证据。具体而言，采用混合查询连接器分离锚定元数据的视觉流与探索性视觉流，并引入轻量级可靠性感知双门控向量调制模块，在噪声输入下自适应调节两者贡献。在大规模真实电商数据集上的全池检索实验表明，TGQ-Former持续优于强连接器基线与端到端多模态大模型，平均提升命中率@100（H@100）6.04%，验证了文本引导视觉编码在鲁棒多模态检索中的有效性。
原文摘要 · Abstract (English)
Multimodal item embeddings are crucial for e-commerce item-to-item (I2I) retrieval, yet real-world product images often contain promotional overlays and background clutter that inject spurious visual cues and degrade retrieval robustness. This issue is particularly pronounced in MLRM-style pipelines, where a frozen vision encoder is connected to an LLM through a lightweight connector that must selectively aggregate visual tokens. We propose Text-Guided Q-Former (TGQ-Former), a text-guided visual representation learning framework that leverages structured metadata as semantic guidance for visual token extraction while preserving complementary visual evidence. Concretely, TGQ-Former employs a hybrid-query connector to disentangle metadata-anchored and exploratory visual streams, and introduces a lightweight reliability-aware dual-gated vector modulation module to adaptively calibrate their contributions under noisy inputs. Experiments on large-scale, real-world e-commerce datasets with full-pool retrieval show that TGQ-Former consistently outperforms strong connector baselines and end-to-end MLLMs. On average, it improves Hit Rate@100 (H@100) by 6.04%, demonstrating the effectiveness of text-guided visual encoding for robust multimodal retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。