用网格化视觉提示提升视频问答中的文字识别精度。
Grid-LOGAT: Grid Based Local and Global Area Transcription for Video Question Answering
- 将视频帧划分为网格，分别提取局部细节与全局信息融合
- 在NExT-QA和STAR-QA上分别达到65.9%和50.11%准确率
- 适合注重隐私保护与定位精度的边缘视频分析场景
本文提出一种基于网格的局部与全局区域文本转录系统（Grid-LoGAT），用于视频问答任务。系统分两阶段运行：首先利用视觉语言模型（VLM）从视频帧中提取文本转录，其次通过大语言模型（LLM）处理问题并生成答案。该设计通过将VLM部署在边缘设备、LLM置于云端，保障图像隐私。为提升转录质量，提出网格化视觉提示机制，从每个网格单元中提取精细局部特征，并融合全局上下文信息。评估结果表明，使用开源模型LLaVA-1.6-7B与Llama-3.1-8B，Grid-LoGAT在NExT-QA和STAR-QA数据集上分别取得65.9%和50.11%的准确率，优于现有方法。在自建的基于定位的问题上，相比非网格版本提升24个百分点。
原文摘要 · Abstract (English)
In this paper, we propose a Grid-based Local and Global Area Transcription (Grid-LoGAT) system for Video Question Answering (VideoQA). The system operates in two phases. First, extracting text transcripts from video frames using a Vision-Language Model (VLM). Next, processing questions using these transcripts to generate answers through a Large Language Model (LLM). This design ensures image privacy by deploying the VLM on edge devices and the LLM in the cloud. To improve transcript quality, we propose grid-based visual prompting, which extracts intricate local details from each grid cell and integrates them with global information. Evaluation results show that Grid-LoGAT, using the open-source VLM (LLaVA-1.6-7B) and LLM (Llama-3.1-8B), outperforms state-of-the-art methods with similar baseline models on NExT-QA and STAR-QA datasets with an accuracy of 65.9% and 50.11% respectively. Additionally, our method surpasses the non-grid version by 24 points on localization-based questions we created using NExT-QA. (This paper is accepted by IEEE ICIP 2025.)
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。