新模型通过时间一致性与变化性建模，提升视频问答准确率
Video Flow as Time Series: Discovering Temporal Consistency and Variability for VideoQA
- 用布朗桥、差异和融合三模块捕捉视频时间连续与突变
- 在多个数据集上显著优于传统Transformer架构
- 适合需要精细时序理解的视频分析任务
视频问答（VideoQA）是一项复杂的多模态任务，要求对视觉内容和时间动态有深入理解。传统Transformer架构虽能融合多模态信息，但常通过位置编码简化时间动态，难以捕捉视频序列中的非线性交互。本文提出时间三元转换器（T3T），通过三个核心组件建模时间一致性与变化性：时间平滑（TS）模块采用布朗桥捕捉连续的时间过渡；时间差异（TD）模块识别并编码视频中显著的时间变化与突变；时间融合（TF）模块将这些时间特征与文本线索结合，促进更深层次的上下文理解与回答准确性。T3T在多个VideoQA基准数据集上进行了广泛测试，结果表明，对时间建模的精细化处理显著提升了视频问答的准确性和深度。
原文摘要 · Abstract (English)
Video Question Answering (VideoQA) is a complex video-language task that demands a sophisticated understanding of both visual content and temporal dynamics. Traditional Transformer-style architectures, while effective in integrating multimodal data, often simplify temporal dynamics through positional encoding and fail to capture non-linear interactions within video sequences. In this paper, we introduce the Temporal Trio Transformer (T3T), a novel architecture that models time consistency and time variability. The T3T integrates three key components: Temporal Smoothing (TS), Temporal Difference (TD), and Temporal Fusion (TF). The TS module employs Brownian Bridge for capturing smooth, continuous temporal transitions, while the TD module identifies and encodes significant temporal variations and abrupt changes within the video content. Subsequently, the TF module synthesizes these temporal features with textual cues, facilitating a deeper contextual understanding and response accuracy. The efficacy of the T3T is demonstrated through extensive testing on multiple VideoQA benchmark datasets. Our results underscore the importance of a nuanced approach to temporal modeling in improving the accuracy and depth of video-based question answering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。