用视觉搜索增强推理,让模型从商品视频中更准提取属性值。
Visual Search Augmented Chain-of-Thought Reasoning for Attribute Value Extraction from Product Videos

- 先聚类找关键帧,再搜相似商品知识补足信息
- 通过多轮交叉推理,使属性预测准确率提升17.91个百分点
- 无需训练,适配任意开源视频模型,适合电商场景
现有视觉属性值提取方法主要依赖静态商品图像,难以捕捉时间动态、多角度视图和细粒度视觉细节。直接应用视频视觉语言模型(VLM)于商品属性提取时性能受限,因缺乏领域知识,而微调又需大量高质量数据与计算资源。为此,我们提出视觉搜索增强的链式思维推理(ViS-CoT),一种无需训练、即插即用的流水线,可应用于任意开源视频VLM以实现电商场景下的视频到文本属性提取。具体而言,ViS-CoT通过视觉聚类识别代表性帧,再利用视觉搜索检索语义相似的商品知识以丰富属性线索;随后,交错式链式思维模块通过从字幕和语音识别生成的视觉对齐辅助文本,迭代优化推理过程;最终,整合信息引导模型实现精准且细粒度的属性预测。在VideoAVE数据集上跨14个商品类别进行的大量实验表明,ViS-CoT持续提升多个前沿视频VLM,在micro-F1上平均提升17.91个百分点。
原文摘要 · Abstract (English)
Existing approaches to visual attribute value extraction (AVE) primarily rely on static product images, failing to capture temporal cues, multi-angle views and fine-grained visual details. Directly applying video vision-language models (VLMs) to product AVE results in limited performance due to the lack of domain knowledge, and fine-tuning them requires extensive high-quality data and substantial computational resources. Thus, we propose visual search augmented chain-of-thought reasoning (ViS-CoT), a training-free, plug-and-play pipeline that can be easily applied to any open-source video VLM for video-to-text AVE in e-Commerce. Specifically, ViS-CoT employs visual clustering to identify representative frames, followed by visual search to retrieve semantically similar product knowledge that can enrich attribute cues. Next, an interleaved CoT reasoning module iteratively refines reasoning through visually-aligned auxiliary texts derived from captioning and automatic speech recognition. Finally, the integrated information guides the model toward accurate and fine-grained attribute predictions. Extensive experiments across 14 product categories on the VideoAVE dataset show that ViS-CoT consistently enhances multiple state-of-the-art video VLMs, achieving an average improvement of 17.91 percentage points in micro-F1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。