让大模型直接处理向量数据，实现跨模态的上下文学习。
Vector-ICL: In-context Learning with Continuous Vector Representations
- 用轻量投影器将向量映射到模型嵌入空间，实现向量上下文学习。
- 在多个任务上超越传统少样本学习和专用模型性能。
- 适合需要跨模态处理向量输入的研究者或开发者使用。
大型语言模型（LLMs）在文本数据上展现出强大的上下文学习（ICL）能力。本文探索这些能力能否扩展至来自不同领域的连续向量，这些向量由黑箱预训练编码器生成。通过轻量级投影器将输入数据对齐至LLM的嵌入空间，我们发现模型能有效处理并从这些投影向量中学习，这一方法称为向量-ICL（Vector-ICL）。实验表明，使用通用语言建模目标预训练投影器即可实现向量-ICL，而任务特定微调可进一步提升性能。在文本重建、数值函数回归、文本分类、摘要生成、分子描述、时间序列分类、图分类及fMRI解码等多任务多模态实验中，向量-ICL通常优于少样本ICL和领域专用模型或微调方法。进一步分析与案例研究显示，大模型具备处理非传统基于标记的向量表示的潜力。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown remarkable in-context learning (ICL) capabilities on textual data. We explore whether these capabilities can be extended to continuous vectors from diverse domains, obtained from black-box pretrained encoders. By aligning input data with an LLM's embedding space through lightweight projectors, we observe that LLMs can effectively process and learn from these projected vectors, which we term Vector-ICL. In particular, we find that pretraining projectors with general language modeling objectives enables Vector-ICL, while task-specific finetuning further enhances performance. In our experiments across various tasks and modalities, including text reconstruction, numerical function regression, text classification, summarization, molecule captioning, time-series classification, graph classification, and fMRI decoding, Vector-ICL often surpasses both few-shot ICL and domain-specific model or tuning. We further conduct analyses and case studies, indicating the potential of LLMs to process vector representations beyond traditional token-based paradigms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。