用几何先验提升语言模型数据效率，16倍少数据达同等精度。
Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA
- 基于语义流形测地线假设，设计隐状态轨迹约束机制
- 在NL-RX-SYNTH上仅用1/16数据达到基线准确率
- 无需多视角增强，适合低资源场景下的大模型训练
大型语言模型遵循可预测的缩放定律——即损失随计算量、数据量和参数量变化的幂律关系。然而这些定律仅描述典型训练过程，而非最优训练。鲜有研究能突破其隐含的数据效率边界，这正是本文核心目标。我们提出测地线假设：词元序列在平滑语义流形上沿测地线运动，因此具有局部线性特性。基于此，我们引入一种新型语义管预测（STP）任务，作为类似JEPA的正则化手段，将隐藏状态轨迹约束在测地线的管状邻域内。该方法无需显式多视图增强即可推广至语言建模。实验证明，该约束提升了信噪比，有效防止推理中轨迹碰撞，保留了多样性。在NL-RX-SYNTH数据集上，使用16倍更少的训练数据即可达到基线准确率，直接违反了Chinchilla型缩放定律中的数据项，证明了合理几何先验可超越单纯规模扩展。代码已开源。
原文摘要 · Abstract (English)
Large Language Models (LLMs) obey consistent scaling laws -- empirical power-law fits that predict how loss decreases with compute, data, and parameters. While predictive, these laws are descriptive rather than prescriptive: they characterize typical training, not optimal training. Surprisingly few works have successfully challenged the data-efficiency bounds implied by these laws -- which is our primary focus. To that end, we introduce the Geodesic Hypothesis, positing that token sequences trace geodesics on a smooth semantic manifold and are therefore locally linear. Building on this principle, we propose a novel Semantic Tube Prediction (STP) task, a JEPA-style regularizer that confines hidden-state trajectories to a tubular neighborhood of the geodesic. STP generalizes JEPA to language without requiring explicit multi-view augmentations. We show this constraint improves signal-to-noise ratio, and consequently preserves diversity by preventing trajectory collisions during inference. Empirically, STP allows LLMs to match baseline accuracy with 16$\times$ less training data on the NL-RX-SYNTH dataset, directly violating the data term of Chinchilla-style scaling laws and demonstrating that principled geometric priors can surpass brute-force scaling. Code is available at https://github.com/galilai-group/llm-jepa#stp.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。