用每秒一张压缩图提升长视频问答效率,结合推理链训练显著提升准确率。
POVQA: Preference-Optimized Video Question Answering with Rationales for Data Efficiency
- 将每秒视频压缩为一张图像,固定上下文长度下保持密集时间覆盖。
- 在自建数据集上,性能从0.212提升至0.550,零样本迁移达64.7%准确率。
- 适合关注长视频多模态推理与高效训练的模型开发者和研究者。
长视频多模态问答需要对视觉证据和对话进行结构化推理,但大视觉语言模型受限于上下文窗口和计算资源。本文提出POVQA,将每秒内容压缩为一张时序聚合图像(1帧/秒),在固定令牌预算下维持密集时间覆盖。使用监督微调(SFT)对Qwen2.5-VL-7B模型进行推理链+答案目标训练,并可选地应用直接偏好优化(DPO)实现偏好对齐。引入ReasonVQA作为诊断数据集,包含12部电影和239个由人工标注的问答+推理链三元组,用于受控分析压缩下的长上下文多模态推理。在ReasonVQA上,SFT将仅使用聚合图像的基线F1从0.212提升至0.550,表明聚合证据与推理链监督是主要增益来源。零样本迁移中,经过SFT+DPO后,POVQA在TVQA上达到64.7%准确率。结果尚属初步:ReasonVQA规模较小,聚合可能丢失精细时间顺序,且DPO效果在不同场景中不一致。代码、数据集及附加定性评估见https://povqa.github.io。
原文摘要 · Abstract (English)
Long-video multimodal question answering requires structured reasoning over visual evidence and dialogue, but Large Vision-Language Models (LVLMs) are constrained by context-window and compute limits. We propose POVQA, which compresses each second into a temporally pooled image (1 fps pooled images) to maintain dense temporal coverage under a fixed token budget. We then train Qwen2.5-VL-7B with supervised fine-tuning (SFT) on rationale+answer targets, and optionally apply Direct Preference Optimization (DPO) for preference alignment. We introduce ReasonVQA as a pilot diagnostic dataset with 12 movies and 239 human-annotated QA+rationale triplets for controlled analysis of long-context multimodal reasoning under compression. On ReasonVQA, SFT improves the best pooled-only baseline from 0.212 to 0.550 F1, showing that pooled evidence plus rationale supervision provides the main performance gains in this setting. In zero-shot transfer, POVQA also reaches 64.7\% on TVQA after SFT+DPO. These results are preliminary: ReasonVQA is small, pooling can lose fine-grained temporal order, and DPO effects are not uniformly positive across settings. Code, dataset, and additional qualitative evaluations are available at \href{https://povqa.github.io}{https://povqa.github.io}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。