发现大模型处理多文字时,拉丁字母有特殊优先地位。
The Latin Substrate: How Language Models Represent and Mediate Script Choice
- 用逐层分析法发现模型内部存在统一的罗马化表征。
- 线性方向可切换输出文字,非拉丁文转拉丁文更稳定。
- 少数晚期注意力头决定文字选择,且跨语言通用。
许多语言使用多种书写系统,要求大语言模型在不同正字法间生成等效内容。现有研究认为模型通过共享潜在表示传递信息,但其内部如何处理文字差异仍不清晰。本文通过分层输出分布分析(logit lens)发现,翻译过程中存在一致的潜在罗马化特征;进一步的表征与机制分析表明,同一语言的不同文字在深层逐渐可分,一个简单线性方向即可切换输出文字,保持语义不变。该方向对未见文字系统具有不对称泛化能力:可稳定将非拉丁文字转为拉丁文字,但反向转换则产生多样非拉丁形式。机制层面,定位到少数晚期注意力头,它们因果性地控制文字选择,且可在无关语言和文字系统间迁移,表明文字路由由语言无关组件实现。两种分析均显示显著方向不对称性:非拉丁文字由局部明确门控产生,而拉丁文字来自网络整体分散贡献。总体表明,模型虽以共享潜在表示组织文字变化,却对拉丁文字存在特权基底。
原文摘要 · Abstract (English)
Many languages are written in multiple scripts, requiring large language models (LLMs) to generate equivalent linguistic content in distinct orthographic forms. While prior work suggests that LLMs route information through shared latent representations, how they internally mediate script variation remains poorly understood. We study this question by first examining per-layer output distributions with the logit lens, which reveals consistent latent romanization during transliteration, and then through representational and mechanistic analyses of script generation. At the representational level, we show that scripts of the same language become increasingly separable across layers and that a simple linear steering direction can flip a model's output script while largely maintaining semantic content. The vector generalizes asymmetrically to writing systems unseen during construction, flipping non-Latin output to Latin reliably, but mapping Latin output into varied non-Latin scripts. At the mechanistic level, we localize a small set of late-layer attention heads that causally mediate script choice. These heads transfer across unrelated languages and writing systems, suggesting that script routing is implemented by language-agnostic components. Across both analyses, we observe a consistent directional asymmetry: non-Latin output is produced by a compact, identifiable gate, while Latin-script output emerges from diffuse contributions across the network. Collectively, our findings hint that LLMs organize script variation around shared latent representations while exhibiting a privileged substrate toward Latin script.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。