用强化学习端到端学分词，效果优于传统方法。
You Can Learn Tokenization End-to-End with Reinforcement Learning
- 用评分函数估计法直接优化离散分词边界
- 在1亿参数规模上优于已有方法，效果更优
- 适合想改进语言模型分词机制的研究者
分词是大语言模型中固定的压缩步骤，尽管整体架构趋向端到端。以往工作尝试用启发式方法或直通估计将分词过程融入模型，但存在局限。本文提出使用评分函数估计法学习分词边界，该方法因直接优化离散分词而具有更强理论保障。实验表明，结合强化学习中的时间折扣等技术可有效降低估计方差，使方法具备可行性。在1亿参数规模下，该方法在定性和定量上均优于已有直通估计方法。
原文摘要 · Abstract (English)
Tokenization is a hardcoded compression step which remains in the training pipeline of Large Language Models (LLMs), despite a general trend towards architectures becoming increasingly end-to-end. Prior work has shown promising results at scale in bringing this compression step inside the LLMs' architecture with heuristics to draw token boundaries, and also attempts to learn these token boundaries with straight-through estimates, which treat the problem of drawing discrete token boundaries as a continuous one. We show that these token boundaries can instead be learned using score function estimates, which have tighter theoretical guarantees due to directly optimizing the problem of drawing discrete token boundaries to minimize loss. We observe that techniques from reinforcement learning, such as time discounting, are necessary to reduce the variance of this score function sufficiently to make it practicable. We demonstrate that the resultant method outperforms prior proposed straight-through estimates, both qualitatively and quantitatively at the $100$ million parameter scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。