将视觉领域的联合嵌入预测架构引入语言模型，显著提升训练效果。
LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures
- 用嵌入空间预测替代输入空间重建，设计新型语言模型训练目标。
- 在多个数据集和模型上超越传统训练方法，且抗过拟合能力更强。
- 适合追求高效、稳定训练的语言模型研究者与实践者。
大型语言模型（LLM）的预训练、微调和评估依赖于输入空间重构与生成能力。然而，在视觉领域已发现嵌入空间训练目标（如联合嵌入预测架构，JEPAs）远优于输入空间目标。语言与视觉训练方式的这一差异引发了一个自然问题：语言模型能否借鉴视觉领域的经验？目前尚无基于JEPA的语言模型，反映了在语言任务中设计此类目标的挑战。本文首次提出 LLM-JEPA，一种适用于预训练与微调的基于JEPA的语言模型训练方案。实验表明，该方法在多种模型（Llama3、OpenELM、Gemma2、Olmo）及多个数据集（NL-RX、GSM8K、Spider、RottenTomatoes）上均显著优于标准训练目标，同时具备更强的抗过拟合能力。代码已开源：https://github.com/rbalestr-lab/llm-jepa。
原文摘要 · Abstract (English)
Large Language Model (LLM) pretraining, finetuning, and evaluation rely on input-space reconstruction and generative capabilities. Yet, it has been observed in vision that embedding-space training objectives, e.g., with Joint Embedding Predictive Architectures (JEPAs), are far superior to their input-space counterpart. That mismatch in how training is achieved between language and vision opens up a natural question: {\em can language training methods learn a few tricks from the vision ones?} The lack of JEPA-style LLM is a testimony of the challenge in designing such objectives for language. In this work, we propose a first step in that direction where we develop LLM-JEPA, a JEPA based solution for LLMs applicable both to finetuning and pretraining. Thus far, LLM-JEPA is able to outperform the standard LLM training objectives by a significant margin across models, all while being robust to overfiting. Those findings are observed across numerous datasets (NL-RX, GSM8K, Spider, RottenTomatoes) and various models from the Llama3, OpenELM, Gemma2 and Olmo families. Code: https://github.com/rbalestr-lab/llm-jepa.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。