通过移除大模型翻译缺陷，提升低资源语言的生成质量
MALT: Mechanistic Ablation of Lossy Translation in LLMs for a Low-Resource Language: Urdu
- 分离内部推理与翻译模块，避免语言转换损失
- 在乌尔都语上性能显著提升，保留文化语境
- 适合关注多语言模型公平性与低资源语言研究者
大语言模型主要在英语数据上训练，导致在低资源语言上表现大幅下降。本研究以乌尔都语为例，探究大模型处理低资源语言的挑战。发现模型在非英语提示下仍以英语进行内部推理，最终层充当翻译器将结果转为目标语言。尽管内部英语表征保持连贯，但翻译过程存在信息丢失，导致输出质量差。通过机制性地移除翻译模块，并使用独立翻译模型处理内部隐式表示，模型在乌尔都语上的性能显著提升，同时更好地保留了输入的文化细节。
原文摘要 · Abstract (English)
LLMs are predominantly trained on English data, which leads to a significant drop in performance on low-resource languages. Understanding how LLMs handle these languages is crucial for improving their effectiveness. This study focuses on Urdu as a use case for exploring the challenges faced by LLMs in processing low-resource languages. LLMs primarily reason in English when prompted in another language, with the final layers acting as translators to convert the English response into the target language. This study finds that even for low-resource languages, the internal latent response of LLMs in English is quite coherent; however, the translation features are lossy and result in poor translations, leading to reduced performance. By mechanistically removing these translation features and using a separate translation model to translate the internal latent response of LLM, the performance of LLMs improves significantly while also preserving the cultural nuances of the input in low-resource languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。