首个评估多模态模型预测未来事件能力的基准,揭示当前模型在听觉视觉预测上仍有显著不足。
FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs
- 构建基于大模型辅助的人机协同流水线,生成跨模态时序推理数据集
- 13个多模态模型在8个领域上最高准确率仅64.8%,语音场景表现最差
- 提出OFF训练策略,提升模型对未来事件的泛化与预测能力
尽管多模态大语言模型(MLLMs)具备强大的多模态感知能力,但其从视听线索中预测未来事件的能力仍鲜受关注,现有基准主要聚焦于回溯性理解。为此,我们提出FutureOmni,首个专用于评估视听环境中多模态未来预测的基准。模型需完成跨模态因果与时间推理,并有效利用内部知识预测未来事件。FutureOmni通过可扩展的LLM辅助、人机协同流程构建,包含919段视频和1,034个多项选择题,覆盖8个主领域。对13个多模态模型和7个纯视频模型的评测显示,当前系统在视听未来预测中表现不佳,尤其在语音密集场景下,最佳准确率为64.8%(Gemini 3 Flash)。为缓解此问题,我们构建了7K样本的指令微调数据集,并提出多模态未来预测(OFF)训练策略。在FutureOmni及主流视听与视频基准上的评估表明,OFF显著提升未来预测性能与泛化能力。代码与数据集已公开(https://github.com/OpenMOSS/FutureOmni,https://huggingface.co/datasets/OpenMOSS-Team/FutureOmni)。
原文摘要 · Abstract (English)
Although Multimodal Large Language Models (MLLMs) demonstrate strong omni-modal perception, their ability to forecast future events from audio-visual cues remains largely unexplored, as existing benchmarks focus mainly on retrospective understanding. To bridge this gap, we introduce FutureOmni, the first benchmark designed to evaluate omni-modal future forecasting from audio-visual environments. The evaluated models are required to perform cross-modal causal and temporal reasoning, as well as effectively leverage internal knowledge to predict future events. FutureOmni is constructed via a scalable LLM-assisted, human-in-the-loop pipeline and contains 919 videos and 1,034 multiple-choice QA pairs across 8 primary domains. Evaluations on 13 omni-modal and 7 video-only models show that current systems struggle with audio-visual future prediction, particularly in speech-heavy scenarios, with the best accuracy of 64.8% achieved by Gemini 3 Flash. To mitigate this limitation, we curate a 7K-sample instruction-tuning dataset and propose an Omni-Modal Future Forecasting (OFF) training strategy. Evaluations on FutureOmni and popular audio-visual and video-only benchmarks demonstrate that OFF enhances future forecasting and generalization. We publicly release all code (https://github.com/OpenMOSS/FutureOmni) and datasets (https://huggingface.co/datasets/OpenMOSS-Team/FutureOmni).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。