定位大模型语言控制层,仅微调5%参数即可保持高语言一致性。
LinguaMap: Which Layers of LLMs Speak Your Language and How to Tune Them?
- 通过分层分析发现语言控制集中在模型末尾层。
- 仅微调末尾3-5%参数,六种语言语言一致性超98%。
- 适合需要高效多语言适配的场景,节省大量计算资源。
尽管进行了多语言预训练,大语言模型在非英语任务中仍面临语言控制难题,即无法以目标语言准确响应。本文识别出两类失效模式:多语言迁移瓶颈(语言正确但任务响应错误)和语言一致性瓶颈(任务响应正确但语言错误)。为此设计了涵盖MMLU、MGSM和XQuAD基准的四场景评估协议,并扩展logit lens分析,逐层追踪语言概率与隐藏状态的跨语言语义相似性。结果揭示模型存在三阶段结构:早期层将输入对齐至共享语义空间,中间层执行任务推理,晚期层驱动语言特异性生成。基于此,提出仅微调最终层的语言控制部分。在Qwen-3-32B和Bloom-7.1B上,该方法在六种语言中实现超过98%的语言一致性,仅需微调3-5%参数,且不降低任务准确率。效果几乎等同于全模型微调(各场景均超98%),但计算成本大幅降低。据我们所知,这是首个利用语言控制层定位实现高效多语言适配的方法。
原文摘要 · Abstract (English)
Despite multilingual pretraining, large language models often struggle with non-English tasks, particularly in language control, the ability to respond in the intended language. We identify and characterize two key failure modes: the multilingual transfer bottleneck (correct language, incorrect task response) and the language consistency bottleneck (correct task response, wrong language). To systematically surface these issues, we design a four-scenario evaluation protocol spanning MMLU, MGSM, and XQuAD benchmarks. To probe these issues with interpretability, we extend logit lens analysis to track language probabilities layer by layer and compute cross-lingual semantic similarity of hidden states. The results reveal a three-phase internal structure: early layers align inputs into a shared semantic space, middle layers perform task reasoning, and late layers drive language-specific generation. Guided by these insights, we introduce selective fine-tuning of only the final layers responsible for language control. On Qwen-3-32B and Bloom-7.1B, this method achieves over 98 percent language consistency across six languages while fine-tuning only 3-5 percent of parameters, without sacrificing task accuracy. Importantly, this result is nearly identical to that of full-scope fine-tuning (for example, above 98 percent language consistency for both methods across all prompt scenarios) but uses a fraction of the computational resources. To the best of our knowledge, this is the first approach to leverage layer-localization of language control for efficient multilingual adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。