通过迭代筛选高置信度示例,提升视频模型在分布外任务中的理解能力。
VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understanding
- 基于相似性与置信度,动态选择并迭代优化上下文示例
- 在多个基准上显著提升分布外视频理解性能,尤其在领域特定场景
- 无需微调,低成本扩展有效上下文长度,适合实际部署
视频大模型虽在多模态理解方面取得进展,但在训练数据稀少的分布外(OOD)任务上性能下降。传统微调成本过高,而现有上下文学习(ICL)受限于视频长序列导致的上下文长度瓶颈。为此,本文提出VideoICL,一种面向分布外视频理解的新型上下文学习框架,包含基于相似性的相关示例选择策略和基于置信度的迭代推理机制。该方法根据示例与任务的相似度排序并选取最优样本,若生成结果置信度低则重新选样并迭代推理,直至获得高置信响应。实验表明,该方法在多个基准上实现显著性能提升,特别是在领域特定场景中表现优异,且无需微调,有效扩展了上下文长度,为更广泛的视频理解应用奠定基础。代码将开源于 https://github.com/KangsanKim07/VideoICL。
原文摘要 · Abstract (English)
Recent advancements in video large multimodal models (LMMs) have significantly improved their video understanding and reasoning capabilities. However, their performance drops on out-of-distribution (OOD) tasks that are underrepresented in training data. Traditional methods like fine-tuning on OOD datasets are impractical due to high computational costs. While In-context learning (ICL) with demonstration examples has shown promising generalization performance in language tasks and image-language tasks without fine-tuning, applying ICL to video-language tasks faces challenges due to the limited context length in Video LMMs, as videos require longer token lengths. To address these issues, we propose VideoICL, a novel video in-context learning framework for OOD tasks that introduces a similarity-based relevant example selection strategy and a confidence-based iterative inference approach. This allows to select the most relevant examples and rank them based on similarity, to be used for inference. If the generated response has low confidence, our framework selects new examples and performs inference again, iteratively refining the results until a high-confidence response is obtained. This approach improves OOD video understanding performance by extending effective context length without incurring high costs. The experimental results on multiple benchmarks demonstrate significant performance gains, especially in domain-specific scenarios, laying the groundwork for broader video comprehension applications. Code will be released at https://github.com/KangsanKim07/VideoICL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。