让视觉语言模型读懂更长文本，提升上下文理解能力。
LongProLIP: A Probabilistic Vision-Language Model with Long Context Text
- 提出细调策略，使模型支持256个文本标记的长上下文。
- 在Urban-1k上显著提升长文本理解能力，优于原模型。
- 适合需要处理长文本的视觉问答与图文匹配任务。
近期提出的概率性视觉语言预训练（ProLIP）模型虽在大规模下实现了概率表征学习，但受限于最大64个上下文长度，无法有效处理更长文本序列。本文提出一种针对ProLIP的细调策略，使其可接受长达256个文本标记的输入。在Urban-1k和DataComp评估套件上的实验表明，所提LongProLIP方法能有效提升长上下文理解能力，同时最小化细调带来的负面影响。我们还观察到长上下文理解能力（以Urban-1k衡量）与通用零样本性能（以DataComp数据集衡量）之间存在权衡关系。代码已开源：https://github.com/naver-ai/prolip。
原文摘要 · Abstract (English)
Recently, Probabilistic Language-Image Pre-Training (ProLIP) has been proposed to tackle the multiplicity issue of vision-language (VL) tasks. Despite their success in probabilistic representation learning at a scale, the ProLIP models cannot handle long context texts longer than 64 context length, which limits their ability to capture rich contextual information from longer text sequences. To address this issue, this paper proposes a fine-tuning strategy for ProLIP to accept longer texts, e.g., 256 text tokens. Experimental results on Urban-1k and the DataComp evaluation suite show that the proposed LongProLIP recipe can improve understanding of long contexts while minimizing the negative effect of fine-tuning.We also observe a trade-off between the long context understanding (measured by Urban-1k) and general zero-shot capability (measured by evaluation datasets by DataComp). Code is available at https://github.com/naver-ai/prolip
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。