让大模型看清字在词中的位置，提升中文纠错等任务效果
Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learning
- 通过反向预测字符训练模型，学习词内字符位置信息
- 在中文纠错任务中，字符定位准确率显著提升
- 不依赖精确位置的任务也能受益，适用性广
分词方法如字节对编码（BPE）虽提升大语言模型（LLMs）的计算效率，但常掩盖词内的字符结构，影响模型对字符位置的判断能力，这对中文拼写纠正（CSC）等任务至关重要。我们提出一种名为词内位置感知（TIPA）的方法，通过在分词器词汇表上进行反向字符预测训练，显著增强模型捕捉词内字符位置的能力。实验表明，TIPA提升了模型在字符位置预测上的准确性，使原始文本中目标字符的定位更精确。此外，在无需精确位置预测的下游任务中，只要涉及字符级信息，TIPA仍能提升性能，验证了其通用性和有效性。
原文摘要 · Abstract (English)
Tokenization methods like Byte-Pair Encoding (BPE) enhance computational efficiency in large language models (LLMs) but often obscure internal character structures within tokens. This limitation hinders LLMs' ability to predict precise character positions, which is crucial in tasks like Chinese Spelling Correction (CSC) where identifying the positions of misspelled characters accelerates correction processes. We propose Token Internal Position Awareness (TIPA), a method that significantly improves models' ability to capture character positions within tokens by training them on reverse character prediction tasks using the tokenizer's vocabulary. Experiments demonstrate that TIPA enhances position prediction accuracy in LLMs, enabling more precise identification of target characters in original text. Furthermore, when applied to downstream tasks that do not require exact position prediction, TIPA still boosts performance in tasks needing character-level information, validating its versatility and effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。