探究多语言大模型是否为每种语言配备专用注意力头。
Do Multilingual LLMs have specialized language heads?
- 通过分析注意力头分布，检验多语言模型是否存在语言特异性结构。
- 移除非目标语言的注意力头后，目标语言性能基本不变。
- 适合需要轻量化部署多语言模型的开发者参考。
多语言大语言模型因其跨语言文本处理与生成能力而广受欢迎。然而，在仅需部分语言支持时，全量部署效率较低。尽管已有研究探讨机器翻译模型是否存在语言特异性或通用性注意力头，但目前尚无针对具备多种任务能力的多语言LLM的相关研究。本文探究多语言LLM是否为每种语言配备专用注意力头，并评估移除非目标语言的注意力头是否会影响目标语言性能。研究发现，移除特定语言的注意力头后，目标语言的表现保持稳定，表明可实现更高效的模型部署，在降低模型复杂度的同时维持高精度。
原文摘要 · Abstract (English)
Multilingual large language models (LLMs) have gained significant popularity for their ability to process and generate text across multiple languages. However, deploying these models in production can be inefficient when only a subset of the supported languages is of interest. There has been some research conducted on identifying whether machine translation models have language-specific or language-agnostic heads, however no research has been conducted for multilingual LLMs, to the best of our knowledge, that as we know are capable of performing diverse tasks beyond just translation. This paper explores whether multilingual LLMs have specialized language attention heads for each language, and investigates the possibility of removing language-specific heads for unwanted languages without degrading performance in the targeted languages. Our findings could inform more efficient deployment strategies for multilingual LLMs, enabling reduced model complexity while maintaining high accuracy for targeted languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。