发现翻译功能向量具语言无关性,跨语言迁移效果显著。
Exploring Language-Agnosticity in Function Vectors: A Case Study in Machine Translation
- 用单向英到其他语言数据提取功能向量
- 向量在多未见语言上提升翻译词排名
- 适合研究模型通用表征与跨语言迁移
功能向量(FVs)是从模型激活中提取的任务向量表示,用于上下文学习。尽管已有研究显示多语言模型表征具有语言无关性,但功能向量是否同样如此仍不明确。本文以机器翻译为例,研究功能向量的语言无关性。在三种仅解码器的多语言大模型上,我们发现从单一英文→目标语言方向提取的翻译功能向量,可有效迁移到其他目标语言,持续提升多个未见语言中正确翻译词的排名。进一步发现,增益最高的词跨越多种语言,且不同方向的翻译功能向量共享大部分顶级注意力头,表明功能向量编码的是主要语言无关的翻译信号,而非特定语言对映射。
原文摘要 · Abstract (English)
Function vectors (FVs) are vector representations of tasks extracted from model activations during in-context learning. While prior work has shown that multilingual model representations can be language-agnostic, it remains unclear whether the same holds for function vectors. We study whether FVs exhibit language-agnosticity, using machine translation as a case study. Across three decoder-only multilingual LLMs, we find that translation FVs extracted from a single English$\to$X direction transfer to other target languages, consistently improving the rank of correct translation tokens across multiple unseen languages. We further find that the highest-gain tokens span multiple languages and that translation FVs across directions share most of their top-ranked heads, indicating that the FV encodes a largely language-agnostic translation signal rather than a language-pair-specific mapping.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。