不联合训练的单语模型也能自发对齐语言表征,靠的是语言本身的结构。
Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?

- 用单语数据训练的模型自动形成可对齐的隐藏表示几何结构
- 仅用一个旋转变换即可跨模型映射平行句的隐状态,准确率高
- 适合做多语言模型拼接、合并或模块化系统设计
多语言语言模型中的跨语言对齐通常归因于联合训练:共享参数、混合语言批次或显式对齐目标。我们探讨了仅在非平行数据上训练的单语模型是否能在无联合训练的情况下学习到可对齐的表征。通过对严格单语的语言模型(如Goldfish系列及不同研究机构独立开发的模型)进行测试,发现三个结果:相关性——这些模型在各层中发展出可对齐的表征几何结构,且随着数据规模、模型规模或语言接近度提升,对齐程度增强;构造性——对平行句使用单一Procrustes旋转即可映射不同模型的隐藏状态;因果性——相同的旋转可传递功能内容;将旋转后的英文残差块插入德语模型,在事实填空任务中多数情况下预测结果会变为源模型的首都。这表明跨语言对齐可源自语言结构及其携带的信息,而非联合训练,为未来模型拼接、合并及由单语组件构建模块化多语言系统提供了实用方向。
原文摘要 · Abstract (English)
Cross-lingual alignment in multilingual language models is typically attributed to joint training: shared parameters, mixed-language batches, or explicit alignment objectives. We ask whether monolingual models trained on non-parallel data learn alignable representations without joint training. By testing on strictly monolingual language models, such as the Goldfish model families and independently developed models from different research labs, we find three results. Correlation: these models develop alignable representational geometry across layers, with alignment strengthening as data scale, model scale, or linguistic proximity increases. Construction: a single Procrustes rotation fit on parallel sentences maps hidden states between models. Causation: the same rotation transfers functional content; patching a rotated English residual into a German model on a factual cloze flips the prediction to the donor's capital in most cases. We confirm that cross-lingual alignment can emerge from the structure of language and the information it carries rather than from joint training, and this points to practical future directions including model stitching, merging, and modular multilingual systems built from monolingual components.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。