通过早层LoRA微调，让斯瓦希里语大模型更好理解中英词汇对应关系。
Targeted Lexical Injection: Unlocking Latent Cross-Lingual Alignment in Lugha-Llama via Early-Layer LoRA Fine-Tuning
- 在模型早期层定位精准的词义对齐，用对比学习优化其输出表现。
- 623组词对的词义相似度从0.3211提升至0.4113，提升28.08%。
- 对未见词对也有效，适合低资源语言大模型的轻量级优化。
大型语言模型在低资源语言（如斯瓦希里语）上的表现常因数据稀缺而受限，核心挑战在于实现稳健的跨语言词义对齐。本文提出目标词注入（TLI）方法，发现斯瓦希里语专属模型Lugha-Llama-8B-wura在早期层（第2层）已具备近乎完美的中英词对对齐能力（平均余弦相似度约0.99998），但该能力在最终输出层仅体现为0.3211的相似度。通过在该最优早层使用低秩适应（LoRA）与对比学习进行微调，实验显示，623个训练词对的输出层相似度从0.3211升至0.4113（+28.08%，p < 1.33×10⁻²⁴⁰）。更重要的是，对63个未见控制词对，相似度从0.3143升至0.4033（+28.32%，p < 7.17×10⁻²⁷），表明该方法能有效保留并传播模型内在的跨语言知识，为低资源语言大模型提供高效、参数高效的对齐优化策略。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their performance in low-resource languages (LRLs), such as Swahili, often lags due to data scarcity and underrepresentation in pre-training. A key challenge is achieving robust cross-lingual lexical alignment, crucial for tasks like translation and cross-lingual information retrieval. This paper introduces Targeted Lexical Injection (TLI), a novel and efficient fine-tuning approach. We first demonstrate that Lugha-Llama-8B-wura, a Swahili-centric LLM, exhibits strong, near-perfect lexical alignment for Swahili-English word pairs in its early internal layers (specifically Layer 2, with ~0.99998 average cosine similarity based on a pilot study), a capability not fully reflected in its final output representations (baseline ~0.32 similarity on our evaluation set). TLI leverages this insight by using Low-Rank Adaptation (LoRA) and a contrastive learning objective to fine-tune the model, specifically targeting embeddings from this empirically identified optimal early layer. Our experiments show that TLI significantly improves the output-level lexical alignment for 623 trained Swahili-English word pairs, increasing average cosine similarity from 0.3211 to 0.4113 (+28.08%, p < 1.33 x 10^-240). More importantly, these improvements generalize remarkably well to 63 unseen control word pairs, with similarity increasing from 0.3143 to 0.4033 (+28.32%, p < 7.17 x 10^-27). These findings suggest TLI enhances the model's ability to preserve and propagate its inherent early-layer cross-lingual knowledge, offering a parameter-efficient and effective strategy for improving lexical alignment in LRL-focused LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。