通过熵估计提升语言模型压缩与泛化能力，实现更高效训练。
Know Your Limits: Entropy Estimation Modeling for Compression and Generalization
- 引入编码器增强的因果解码器架构，提升训练效率。
- 在普通硬件上实现比因果Transformer更高的压缩率。
- 按词元估算熵，指导训练以提升模型泛化性能。
语言预测受语言内在信息熵限制，任何语言模型的准确率存在上限，相应地也存在语言压缩的下限。当前最高效的压缩算法是基于因果（下一个词预测）的大语言模型，但利用这些模型估算语言熵在计算上仍不可行。本文提出编码器增强的因果解码器架构，具有更优的训练效率，在适度硬件上即可实现优于因果Transformer的压缩效果。我们展示了如何实现逐词元的熵估计，并证明：训练目标趋近于训练数据熵值的模型，其泛化能力必然优于仅最小化损失而忽略熵的模型。实验证明，训练时趋近但不超越估计熵值的因果模型，泛化性能显著优于未考虑熵的模型。
原文摘要 · Abstract (English)
Language prediction is constrained by informational entropy intrinsic to language, such that there exists a limit to how accurate any language model can become and equivalently a lower bound to language compression. The most efficient language compression algorithms today are causal (next token prediction) large language models, but the use of these models to form accurate estimates of language entropy is currently computationally infeasible. We introduce encoder-augmented causal decoder model architectures that exhibit superior training efficiency characteristics and achieve higher compression than causal transformers even when trained on modest hardware. We demonstrate how entropy estimates can be obtained on a per-token basis, and show that the generalization of models trained to approach the entropy of their training data necessarily exceeds the generalization of models trained to minimize loss beyond this value. We show empirically that causal models trained to approach but not exceed estimated per-token entropies exhibit greater generalization than models trained without taking entropy into account.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。