用预测未来信息提升实时音视频事件解析准确率
PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling
- 通过预测未来多模态信号增强在线上下文理解
- 在UnAV-100和LLP数据集上显著超越现有方法
- 参数量少，适合部署于实时系统
音视频事件解析对理解多模态视频内容至关重要，但现有方法通常依赖完整视频的离线处理且模型庞大，难以实时应用。本文提出在线音视频事件解析（On-AVEP）新范式，通过逐帧分析连续输入视频流实现事件识别。该任务要求模型具备两项关键能力：(1) 准确的在线推理能力，以应对上下文不明确的实时场景；(2) 实时效率，在高性能与计算资源之间取得平衡。为此，我们提出预测未来建模（PreFM）框架，包含：(a) 预测性多模态未来建模，用于推断并整合有益的未来音视频线索，增强上下文理解；(b) 无模态依赖的鲁棒表征与焦点时间优先机制，提升精度与泛化能力。在UnAV-100和LLP数据集上的大量实验表明，PreFM显著优于当前最优方法，且参数量大幅减少，为实时多模态视频理解提供了新思路。代码已开源。
原文摘要 · Abstract (English)
Audio-visual event parsing plays a crucial role in understanding multimodal video content, but existing methods typically rely on offline processing of entire videos with huge model sizes, limiting their real-time applicability. We introduce Online Audio-Visual Event Parsing (On-AVEP), a novel paradigm for parsing audio, visual, and audio-visual events by sequentially analyzing incoming video streams. The On-AVEP task necessitates models with two key capabilities: (1) Accurate online inference, to effectively distinguish events with unclear and limited context in online settings, and (2) Real-time efficiency, to balance high performance with computational constraints. To cultivate these, we propose the Predictive Future Modeling (PreFM) framework featured by (a) predictive multimodal future modeling to infer and integrate beneficial future audio-visual cues, thereby enhancing contextual understanding and (b) modality-agnostic robust representation along with focal temporal prioritization to improve precision and generalization. Extensive experiments on the UnAV-100 and LLP datasets show PreFM significantly outperforms state-of-the-art methods by a large margin with significantly fewer parameters, offering an insightful approach for real-time multimodal video understanding. Code is available at https://github.com/XiaoYu-1123/PreFM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。