让图文预训练模型更好理解长文本，提升检索效果。
LoTLIP: Improving Language-Image Pre-training for Long Text Understanding
- 用长描述重标注数据，增强长文本理解能力。
- 引入角标记聚合信息，保持短文本理解性能。
- 在1亿对图文数据上验证，长文本检索更优。
实际应用中对长文本理解需求迫切，但多数语言-图像预训练（LIP）模型难以胜任。我们实证发现，问题根源在于训练图像通常配以短标题，导致部分词汇被显著词掩盖。初始尝试用长标题重新标注数据，但会损害短文本理解能力（如图像分类）。为此，通过引入角标记聚合多样化文本信息，使模型在维持原有短文本理解水平的同时，大幅提升长文本理解能力。进一步研究发现，随着标题变长，性能与效率之间存在明显权衡。我们在自建的1亿对长标题导向图文数据集上验证了方法有效性，其在长文本-图像检索任务中表现卓越。
原文摘要 · Abstract (English)
Understanding long text is of great demands in practice but beyond the reach of most language-image pre-training (LIP) models. In this work, we empirically confirm that the key reason causing such an issue is that the training images are usually paired with short captions, leaving certain tokens easily overshadowed by salient tokens. Towards this problem, our initial attempt is to relabel the data with long captions, however, directly learning with which may lead to performance degradation in understanding short text (e.g., in the image classification task). Then, after incorporating corner tokens to aggregate diverse textual information, we manage to help the model catch up to its original level of short text understanding yet greatly enhance its capability of long text understanding. We further look into whether the model can continuously benefit from longer captions and notice a clear trade-off between the performance and the efficiency. Finally, we validate the effectiveness of our approach using a self-constructed large-scale dataset, which consists of 100M long caption oriented text-image pairs. Our method demonstrates superior performance in long-text-image retrieval tasks. The project page is available at https://wuw2019.github.io/lot-lip.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。