揭示大模型多语言表征机制，解释为何英语表现更好。
Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders
- 用跨层译码器分析模型内部表征，发现共享多语言特征。
- 后期层依赖少量高频特征解码语言，可干预替换语言。
- 英语优势源于词元化偏见和后期特征弱化，微调可改善。
多语言大模型能处理多种语言，但其内部表征机制仍不清晰。它们是否形成共享的多语言表征并配合语言特定解码？若是，为何性能偏向主流训练语言？我们通过在不同多语言混合数据上训练模型，并利用跨层译码器（CLTs）与归因图分析内部机制。结果表明：模型在不同语言间使用高度相似的特征，语言特定解码出现在后期层。即使不包含英语训练，仍存在相同的多语言共享空间结构。解码依赖于最终层中一组高频特征，这些特征从早期层线性编码语言身份。干预这些特征可抑制一种语言并替换为另一种。为解释非英语语言性能下降，我们进行模型差分实验：性能不足源于晚期特征弱、中间层聚类弱，以及词元化对英语的偏向导致早期层专用于词汇重组。微调增强了这些特征及其关联，改善了词元组装与语言特异性解码，提供了多语言差距的机制解释。相关模型与CLTs可在HuggingFace获取，代码地址：https://github.com/abirharrasse/MultilingualCLTs。
原文摘要 · Abstract (English)
Multilingual Large Language Models (LLMs) can process many languages, yet how they internally represent this diversity remains unclear. Do they form shared multilingual representations with language-specific decoding, and if so, why does performance favor the dominant training language? To address this, we train models on different multilingual mixtures and analyze their internal mechanisms using Cross-Layer Transcoders (CLTs) and Attribution Graphs. Our results reveal multilingual shared representations: the model employs highly similar features across languages, while language-specific decoding emerges in later layers. Training models without English shows identical multilingual shared space structures. Decoding relies partly on a small set of high-frequency features in the final layers, which linearly encode language identity from early layers. Intervening on these features allows one language to be suppressed and another substituted. Finally, to explain non-English failures, we perform a Model-Diffing experiment: underperformance arises from dim late-layer features, weak middle-layer clusters, and tokenizer bias toward English that forces early layers to specialize in word reassembly. Finetuning strengthens these features and their links, improving token assembly and language-specific decoding, providing a mechanistic explanation for multilingual gaps. Our models and CLTs are available at https://huggingface.co/collections/CausalNLP/multilingual-clts and https://huggingface.co/collections/CausalNLP/multilingual-gpt2-models. Our code is available at: https://github.com/abirharrasse/MultilingualCLTs
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。