用模拟人类语言习得机制的预训练，让大模型更高效学语言。
Language Acquisition Device in Large Language Models
- 在合成语言MP-STRUCT上做500步预预训练，模拟人类语言先天结构约束。
- 模型在词元效率上媲美强基准，且对不合理语言有更强抵抗力。
- 关键在于降低依赖歧义的功能标记，不只看表达能力，更要看可解析性。
大型语言模型（LLMs）的数据效率远低于人类。为缩小这一差距，已有研究提出在合成语言上进行预预训练（PPT），强调高度表达性的形式语言，如k-Shuffle Dyck。受语言习得装置（LAD）假说启发，我们提出基于LAD思想的PPT：在MP-STRUCT上预预训练，该形式语言通过MERGE、AGREE和MOVE编码层级组合、特征依赖与长距离移位。仅需500步的简短预预训练，即在词元效率上达到强形式语言基线水平，并赋予模型类似人类对结构不合理语言（如REVERSE）的抗性。分析简化版本发现，尽管MP-STRUCT CORE不可由C-RASP定义（限制Transformer表达力的形式边界），其表现仍优于k-Shuffle Dyck，挑战了此前认为有效PPT语言必须兼具层级表达力与电路可学习性的假设。结果表明，功能地标（减少依赖歧义）是关键驱动力，说明有效PPT设计不仅依赖表达力，还取决于依赖解析的可及性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) remain substantially less data-efficient than humans. Pre-pretraining (PPT) on synthetic languages has been proposed to close this gap, with prior work emphasizing highly expressive formal languages such as $k$-Shuffle Dyck. Inspired by the Language Acquisition Device (LAD) hypothesis, which posits that innate constraints preemptively restrict the learner's hypothesis space to natural-language-like structure, we propose LAD-inspired PPT: pre-pretraining on MP-STRUCT, a formal language whose strings encode hierarchical composition, feature-based dependencies, and long-distance displacement via MERGE, AGREE, and MOVE. A brief 500-step PPT with MP-STRUCT matches strong formal-language baselines in token efficiency while additionally imparting a human-like resistance to structurally implausible languages (e.g., REVERSE). Analyzing simplified variants, we find that MP-STRUCT CORE outperforms $k$-Shuffle Dyck despite not being definable in C-RASP (a formal bound on transformer expressivity), challenging the prior hypothesis that effective PPT languages must be both hierarchically expressive and circuit-theoretically learnable. We show that functional landmarks, which reduce dependency resolution ambiguity, are a key driver, suggesting that effective PPT design depends not only on expressivity but also on the accessibility of dependency resolution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。