arXiv:2608.26576cs.CL2026-08

双语预训练会改变模型对英语概念的表征,即使输入一致。

Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations

论文配图:Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations
图 1 · 摘自论文原文
  • 40个相同架构的模型对比单语与双语训练效果。
  • 双语条件下英语概念位置差异超过随机种子影响。
  • 中间层上下文状态受另一语言影响最明显,适合研究多语表征。

一个概念在不同语言中可能携带不同关联,而现代语言模型在预训练时通常同时学习英语和其他语言。然而,现有模型因训练语料、计算资源、架构和随机种子不同,难以区分单一语言如何影响英语概念的表征。本研究通过控制实验,使用40个参数量为310M的解码器仅模型,共享架构、分词器、训练流程和英语数据源。每个双语条件额外加入一种语言(共8种),并通过4组对照分别控制英语暴露量、总训练量和英语文档重叠。每对模型用3,000个共通英语词对齐后,测量1,000个保留英语概念在50个固定语义对比轴上的位置(如红-白)。在32组对比中,双语与单语条件间英语概念位置差异,大于同为单语但随机种子不同的模型间差异。该差异在上下文状态中更显著,且在中间层达到峰值。表明即便显式对齐英语输入表示,与英语共训的语言仍会改变模型对英语概念的内部表征。

原文摘要 · Abstract (English)

A concept can carry different associations across languages, while modern language models learn English alongside many other languages during pretraining. Yet comparisons among existing models cannot easily isolate how any one language changes the way these models represent English concepts because their training corpora, compute, architectures, and random seeds all differ. We study this question through a controlled experiment with 40 matched 310M-parameter decoder-only models that share an architecture, tokenizer, training recipe, and English data source. Each bilingual condition adds one of eight languages, while four experimental comparisons separately account for English exposure, total training, and English-document overlap. We align each model pair using 3,000 common English words, then measure where 1,000 held-out English concepts fall along 50 fixed semantic contrasts, such as red versus white. Across 32 experimental comparisons, English concept positions differ more between bilingual and English-only conditions than between English-only runs with different random seeds. These differences are larger in contextual states than in token embeddings and peak in middle layers. The language learned alongside English can therefore change how a model represents English concepts even when its English input representations are explicitly aligned.

多语言表征预训练语义差异

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。