测试大模型能否跨语言回忆训练数据中的文本，发现英文记忆可迁移到多种语言。
OWL: Probing Cross-Lingual Recall of Memorized Texts via World Literature
- 构建跨语言对齐文本数据集OWL，覆盖10种语言共31.5万条书摘。
- GPT-4o在新译文中仍能正确识别作者和书名69%、实体名6%。
- 即使打乱或遮蔽文本，模型仍保持一定回忆能力，体现强跨语言记忆特性。
大型语言模型（LLMs）已知会记忆并召回其预训练数据中的英文文本，但这种能力是否能推广至非英语语言或跨语言迁移尚不明确。本文研究了多语言与跨语言记忆现象，探究一种语言中记忆的内容（如英文）在翻译后能否被召回。为此，我们引入了OWL数据集，包含20本书籍在10种语言中的31.5万条对齐文段，涵盖英文原文、官方译本（越南语、西班牙语、土耳其语）以及六种低资源语言的新译本（塞索托语、约鲁巴语、迈提利语、马达加斯加语、茨瓦纳语、夏威夷语）。通过三项任务评估不同模型家族和规模的记忆能力：（1）直接探测，要求模型识别书籍标题与作者；（2）名字填空，预测被遮蔽的角色名；（3）前缀探测，生成后续内容。结果表明，模型在跨语言情境下仍能持续回忆内容，即便目标语言未在预训练中出现。例如，GPT-4o在新译文中识别作者和书名的成功率达69%，预测被遮蔽实体为6%。扰动操作（如遮蔽字符、词序打乱）仅使直接探测准确率下降7%（针对官方译本）。研究揭示了跨语言记忆的广泛存在，并提供了模型间差异的洞察。
原文摘要 · Abstract (English)
Large language models (LLMs) are known to memorize and recall English text from their pretraining data. However, the extent to which this ability generalizes to non-English languages or transfers across languages remains unclear. This paper investigates multilingual and cross-lingual memorization in LLMs, probing if memorized content in one language (e.g., English) can be recalled when presented in translation. To do so, we introduce OWL, a dataset of 31.5K aligned excerpts from 20 books in ten languages, including English originals, official translations (Vietnamese, Spanish, Turkish), and new translations in six low-resource languages (Sesotho, Yoruba, Maithili, Malagasy, Setswana, Tahitian). We evaluate memorization across model families and sizes through three tasks: (1) direct probing, which asks the model to identify a book's title and author; (2) name cloze, which requires predicting masked character names; and (3) prefix probing, which involves generating continuations. We find that LLMs consistently recall content across languages, even for texts without direct translation in pretraining data. GPT-4o, for example, identifies authors and titles 69% of the time and masked entities 6% of the time in newly translated excerpts. Perturbations (e.g., masking characters, shuffling words) modestly reduce direct probing accuracy (7% drop for shuffled official translations). Our results highlight the extent of cross-lingual memorization and provide insights on the differences between the models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。