用推测解码加速视频问答，兼顾速度与准确率
FastV-RAG: Towards Fast and Fine-Grained Video QA with Retrieval-Augmented Generation
- 轻量模型先生成答案候选，重模型验证优化，降低推理延迟
- 通过相似度过滤提升知识检索中的实体对齐，准确率显著提升
- 适合需要快速响应且依赖外部知识的视频问答场景
视觉语言模型在视觉推理方面表现优异，但在整合外部知识方面仍存在困难。检索增强生成（RAG）是潜在解决方案，但现有方法效率低且常影响答案质量。为此，我们提出 VideoSpeculateRAG，一种基于VLM的高效RAG框架，核心包含两项创新：一是引入推测解码流程——轻量级草稿模型快速生成多个答案候选，再由更精准的重型模型验证并优化，显著降低推理延迟而不牺牲正确性；二是识别出检索知识中错误实体识别这一主要误差来源，并通过简单有效的相似度过滤策略改善实体对齐，提升整体答案准确性。实验表明，VideoSpeculateRAG在保持或超越标准RAG准确率的同时，推理速度提升约2倍。该框架展示了结合推测解码与检索增强推理，在复杂、知识密集型多模态任务中提升效率与可靠性的潜力。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) excel at visual reasoning but still struggle with integrating external knowledge. Retrieval-Augmented Generation (RAG) is a promising solution, but current methods remain inefficient and often fail to maintain high answer quality. To address these challenges, we propose VideoSpeculateRAG, an efficient VLM-based RAG framework built on two key ideas. First, we introduce a speculative decoding pipeline: a lightweight draft model quickly generates multiple answer candidates, which are then verified and refined by a more accurate heavyweight model, substantially reducing inference latency without sacrificing correctness. Second, we identify a major source of error - incorrect entity recognition in retrieved knowledge - and mitigate it with a simple yet effective similarity-based filtering strategy that improves entity alignment and boosts overall answer accuracy. Experiments demonstrate that VideoSpeculateRAG achieves comparable or higher accuracy than standard RAG approaches while accelerating inference by approximately 2x. Our framework highlights the potential of combining speculative decoding with retrieval-augmented reasoning to enhance efficiency and reliability in complex, knowledge-intensive multimodal tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。