针对在线课程视频，构建了带时间戳的问答数据集与低延迟跨模态检索方法。
CourseTimeQA: A Lecture-Video Benchmark and a Latency-Constrained Cross-Modal Fusion Method for Timestamped QA
- 用冻结编码器+轻量投影和浅层注意力融合语音与图像特征
- 在单卡A100上实现1.55秒平均端到端延迟，性能优于强基线
- 适合教育场景下实时精准定位视频内容的开发者使用
本文研究在单张GPU资源限制下的教育讲座视频时间戳问答任务。给定自然语言问题，系统需检索相关的时间片段并生成有依据的答案。提出CourseTimeQA数据集（涵盖6门课程，总计52.3小时视频、902个问题），以及一种轻量级、低延迟的跨模态检索方法CrossFusion-RAG。该方法结合冻结编码器、512→768视觉投影、对齐语音与帧的浅层查询无关交叉注意力，并引入时序一致性正则化，辅以小型交叉注意力重排序器。在CourseTimeQA上，CrossFusion-RAG相较强基线BLIP-2检索器，nDCG@10提升0.10，MRR提升0.08，且在单张A100上达到约1.55秒中位端到端延迟。对比多种基准方法（如零样本CLIP多帧池化、CLIP+交叉编码器重排序+MMR等），均在相同硬件与索引条件下评估。报告了对ASR噪声（WER四分位）的鲁棒性、时间定位诊断结果及完整训练调参细节，支持可复现比较。
原文摘要 · Abstract (English)
We study timestamped question answering over educational lecture videos under a single-GPU latency/memory budget. Given a natural-language query, the system retrieves relevant timestamped segments and synthesizes a grounded answer. We present CourseTimeQA (52.3 h, 902 queries across six courses) and a lightweight, latency-constrained cross-modal retriever (CrossFusion-RAG) that combines frozen encoders, a learned 512->768 vision projection, shallow query-agnostic cross-attention over ASR and frames with a temporal-consistency regularizer, and a small cross-attentive reranker. On CourseTimeQA, CrossFusion-RAG improves nDCG@10 by 0.10 and MRR by 0.08 over a strong BLIP-2 retriever while achieving approximately 1.55 s median end-to-end latency on a single A100. Closest comparators (zero-shot CLIP multi-frame pooling; CLIP + cross-encoder reranker + MMR; learned late-fusion gating; text-only hybrid with cross-encoder reranking and its MMR variant; caption-augmented text retrieval; non-learned temporal smoothing) are evaluated under matched hardware and indexing. We report robustness across ASR noise (WER quartiles), diagnostics for temporal localization, and full training/tuning details to support reproducible comparison.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。