用语言桥接视觉与文本，让模型零样本理解真实场景。
Vision-Language Integration for Zero-Shot Scene Understanding in Real-World Environments
- 融合预训练视觉模型与语言模型，对齐视觉与语义空间。
- 在多个数据集上实现最高18%的识别准确率提升。
- 适合需要泛化能力的零样本场景理解任务。
真实世界中零样本场景理解面临自然场景复杂多变的挑战，模型需在无标注样本情况下识别新物体、动作和上下文。本文提出一种视觉-语言融合框架，统一预训练视觉编码器（如CLIP、ViT）与大语言模型（如GPT架构），通过自然语言实现视觉与文本模态的语义对齐。该方法将视觉输入与文本提示嵌入共享空间，经多模态融合与推理层进行上下文解析。在Visual Genome、COCO、ADE20K及自建真实场景数据集上的实验表明，所提系统在物体识别、活动检测和场景描述任务上显著优于现有零样本模型。性能提升最高达18%的top-1准确率，并在语义连贯性指标上取得明显增益，验证了跨模态对齐与语言引导在提升真实场景泛化能力中的有效性。
原文摘要 · Abstract (English)
Zero-shot scene understanding in real-world settings presents major challenges due to the complexity and variability of natural scenes, where models must recognize new objects, actions, and contexts without prior labeled examples. This work proposes a vision-language integration framework that unifies pre-trained visual encoders (e.g., CLIP, ViT) and large language models (e.g., GPT-based architectures) to achieve semantic alignment between visual and textual modalities. The goal is to enable robust zero-shot comprehension of scenes by leveraging natural language as a bridge to generalize over unseen categories and contexts. Our approach develops a unified model that embeds visual inputs and textual prompts into a shared space, followed by multimodal fusion and reasoning layers for contextual interpretation. Experiments on Visual Genome, COCO, ADE20K, and custom real-world datasets demonstrate significant gains over state-of-the-art zero-shot models in object recognition, activity detection, and scene captioning. The proposed system achieves up to 18% improvement in top-1 accuracy and notable gains in semantic coherence metrics, highlighting the effectiveness of cross-modal alignment and language grounding in enhancing generalization for real-world scene understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。