用强化学习让大模型学会从上下文学翻译新语言
Reinforcement Learning Elicits Contextual Learning of Unseen Language Translation
- 用字符级翻译分数做奖励，训练模型从上下文提取语言知识
- 在未见过的语言上表现优于传统提示学习和微调方法
- 适合研究如何让模型通过上下文自学语言的学者
以往工作表明，大语言模型可通过持续训练或在上下文中编码语法书来翻译未见或低资源语言。但这些方法通常会过度拟合特定语言，测试时零样本迁移能力有限。为实现对极端低资源语言的大规模翻译，我们主张大模型应掌握利用上下文语言知识的元技能，而非记忆具体语言。本文提出一种基于强化学习（RL）的未见语言翻译方法，使用表面级翻译指标（chrF）作为奖励。实证结果表明，尽管奖励设计轻量，但经训练的模型能有效从上下文中提取并应用相关语言信息，在完全未见语言上的翻译表现优于上下文学习或监督微调。分析显示，基于结果的强化学习可超越数学与编程等传统推理任务，成为从上下文中学习语言的通用范式。
原文摘要 · Abstract (English)
Prior work has shown that large language models (LLMs) can translate unseen or low-resource languages by undergoing continued training or even by encoding a grammar book in their context. However, both methods typically overfit specific languages, with limited zero-shot transfer at test time. To translate extremely low-resource languages at scale, we argue that LLMs must acquire the meta-skill of utilizing in-context linguistic knowledge rather than memorizing specific languages. In this paper, we propose a reinforcement learning (RL) approach to unseen language translation given rich linguistic context, using a surface-level translation metric (chrF) as the reward. Empirically, despite the lightweight reward, our RL-trained models effectively extract and apply relevant linguistic information from the provided context, leading to better translations on completely unseen languages than in-context learning or supervised fine-tuning. Our analyses suggest that outcome-based RL can extend beyond conventional reasoning tasks like math and coding to serve as a recipe for language learning from context.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。