用汉字候选增强预训练，解决韩语同音词歧义问题
HanjaBridge: Resolving Semantic Ambiguity in Korean LLMs via Hanja-Augmented Pre-Training
- 引入汉字候选池，让模型学习上下文消歧
- 在KoBALT上提升21%性能，跨语言迁移效果显著
- 推理时无需汉字，效率高适合实际应用
大型语言模型在韩语等低资源语言上表现不佳，部分原因在于同音汉字词在韩文书写中无法区分。为此，我们提出HanjaBridge，一种嵌入持续预训练框架的语义注入方法。不同于将词语固定映射为单一汉字，HanjaBridge为同形词提供所有可能的汉字候选，促使模型学习上下文消歧能力。该过程结合词元级知识蒸馏，防止灾难性遗忘。实验表明，HanjaBridge显著提升韩语理解能力，在KoBALT基准上实现21%的相对性能提升。通过共享汉字强化韩语与汉语间的语义对齐，观察到显著的正向跨语言迁移。此外，即使推理时省略汉字增强，性能仍保持不变，确保无额外运行开销。
原文摘要 · Abstract (English)
Large language models (LLMs) often show poor performance in low-resource languages like Korean, partly due to unique linguistic challenges such as homophonous Sino-Korean words that are indistinguishable in Hangul script. To address this semantic ambiguity, we propose HanjaBridge, a novel meaning-injection technique integrated into a continual pre-training (CPT) framework. Instead of deterministically mapping a word to a single Hanja (Chinese character), HanjaBridge presents the model with all possible Hanja candidates for a given homograph, encouraging the model to learn contextual disambiguation. This process is paired with token-level knowledge distillation to prevent catastrophic forgetting. Experimental results show that HanjaBridge significantly improves Korean language understanding, achieving a 21\% relative improvement on the KoBALT benchmark. Notably, by reinforcing semantic alignment between Korean and Chinese through shared Hanja, we observe a strong positive cross-lingual transfer. Furthermore, these gains persist even when Hanja augmentation is omitted at inference time, ensuring practical efficiency with no additional run-time cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。