用字节适配器让通义千问模型更好理解哈萨克语
KazByte: Adapting Qwen models to Kazakh via Byte-level Adapter
- 通过字节级适配器绕过传统分词，直接处理原始字节数据
- 在哈萨克语基准测试中表现接近原版通义千问7B模型
- 适合低资源语言迁移，尤其对哈萨克语等小语种有实用价值
大型语言模型将哈萨克语文本分割成的词元数量远超同等长度的英语，因为其分词器是为高资源语言设计的。这种分词开销导致计算量增加、有效上下文窗口缩短，并削弱了模型对哈萨克语形态的捕捉能力。我们提出绕过分词器，通过一个小型适配器将原始字节输入冻结的Qwen2.5-7B模型内部语言中。适配器训练完成后，仅微调Qwen的注意力层以适应哈萨克语文本。核心假设是：先训练接口，再调整模型，能实现或超越原版Qwen2.5-7B在标准哈萨克语基准上的性能。本文描述了ByteKaz架构与训练流程，实证验证正在进行中，当前版本旨在记录设计与假设。
原文摘要 · Abstract (English)
Large language models fragment Kazakh text into many more tokens than equivalent English text, because their tokenizers were built for high-resource languages. This tokenizer tax inflates compute, shortens the effective context window, and weakens the model's grip on Kazakh morphology. We propose to bypass the tokenizer entirely by feeding raw bytes through a small adapter that learns to speak the internal language of a frozen Qwen2.5-7B. Once the adapter is trained, we freeze it and fine-tune only the attention layers of Qwen on Kazakh text. Our central hypothesis is that this two-stage process -- first teach the interface, then adapt the model -- should match or exceed the accuracy of the original Qwen2.5-7B on standard Kazakh benchmarks. This report describes the ByteKaz architecture and training protocol. Empirical validation is ongoing; this version stakes the design and hypotheses for the record.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。