基于低对数概率秩特性，实现对现代大模型的可证明学习。
Provably Learning from Modern Language Models via Low Logit Rank
- 提出通过低对数概率秩结构建模大语言模型行为
- 设计高效算法，从查询中学习任意近似低秩模型
- 适用于接口查询场景，为大模型学习提供首个端到端保障
尽管现代语言模型及其内部机制极为复杂，但近期研究（Golowich, Liu & Shetty; 2025）通过观察发现，这些模型的对数概率矩阵在实践中近似低秩。这意味着，在给定某些词元序列条件下，模型对各类词元的对数概率构成的矩阵可用低秩矩阵良好逼近。本文聚焦于如何利用该结构获得可证明的学习保证。由于低对数概率秩模型能编码难以学习的分布（如带噪声的奇偶性），我们研究一种反映常见API访问模式的对数概率查询学习模型。主要成果是：提出一种高效算法，可从查询中学习任意近似低秩模型。我们强调，该结构假设与现代语言模型的实际行为高度一致，因此本结果首次为可能刻画现代语言模型的生成模型提供了端到端学习保证。
原文摘要 · Abstract (English)
While modern language models and their inner workings are incredibly complex, recent work (Golowich, Liu & Shetty; 2025) has proposed a simple and potentially tractable abstraction for them through the observation that empirically, these language models all seem to have approximately low logit rank. Roughly, this means that a matrix formed by the model's log probabilities of various tokens conditioned on certain sequences of tokens is well approximated by a low rank matrix. In this paper, our focus is on understanding how this structure can be exploited algorithmically for obtaining provable learning guarantees. Since low logit rank models can encode hard-to-learn distributions such as noisy parities, we study a query learning model with logit queries that reflects the access model for common APIs. Our main result is an efficient algorithm for learning any approximately low logit rank model from queries. We emphasize that our structural assumption closely reflects the behavior that is empirically observed in modern language models. Thus, our result gives what we believe is the first end-to-end learning guarantee for a generative model that plausibly captures modern language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。