让翻译评估懂文化:用强化学习教会模型识别网络梗的真正含义
When Metrics Reward the Worst Translations: Internalizing Cultural Reasoning for Social Media Translation Evaluation

- 把文化注释嵌入模型推理过程,用渐进式掩码训练实现自主文化判断
- 在1444条社交文本上,小模型效果接近大模型,准确率达45.22%
- 适合需要精准理解网络语言的翻译系统优化与评测场景
基于通用语料库训练的自动翻译质量评估指标在社交媒体内容上系统性失效,因其无法捕捉富含文化色彩的表达(如网络俚语、谐音密码、平台特有习语),而仅依赖表面词序模式。我们通过实证分析发现,COMET、XCOMET 和 BERTScore 等标准指标与人类文化判断的相关性接近零或为负,甚至出现评分随翻译质量下降而上升的严重倒置现象。大型语言模型裁判也表现不佳:Qwen3-235B 的 Cohen's kappa 仅为 0.162,表明瓶颈并非推理能力,而是缺乏领域特定的文化知识。为此,我们提出 CuRIL 框架——一种强化学习方法,将文化标注注入模型推理路径,通过词元级损失掩码排除其对策略梯度的影响,并随训练逐步降低注入概率,促使模型自主形成文化判断能力。在包含 1,444 条人工标注的社交媒体翻译基准上,使用 CuRIL 训练的 Qwen3-8B 达到 Cohen's kappa 0.370 与精确匹配率 45.22%,逼近拥有 30 倍参数的 Gemini-3.1-Pro,且优于高达 235B 规模的模型。进一步验证表明,该裁判能提供可靠奖励信号,使下游翻译优化后低质译文率下降超 20 个百分点。
原文摘要 · Abstract (English)
Automatic translation quality metrics trained on general-domain corpora systematically fail on social media content, where communicative intent is encoded in culturally loaded expressions (internet slang, homophonic ciphers, and platform-specific idioms) rather than surface token patterns. We conduct a systematic empirical analysis demonstrating that standard metrics including COMET, XCOMET, and BERTScore exhibit near-zero or negative correlation with human cultural judgments, and even display a severity inversion in which scores increase as translation quality deteriorates. We further show that this failure extends to large language model judges: Qwen3-235B achieves Cohen's kappa of only 0.162, revealing that the bottleneck is not reasoning capacity but cultural grounding: models lack the domain-specific cultural knowledge needed to identify which aspects of a translation require scrutiny. To address this, we propose CuRIL, a reinforcement learning framework that internalizes cultural reasoning: cultural annotations are prepended inside the model's reasoning, excluded from policy gradients via a token-level loss mask, and injected with a probability that decays to zero over training, progressively forcing autonomous cultural judgment. On a 1,444-sample human-annotated social media translation benchmark, Qwen3-8B trained with CuRIL achieves Cohen's kappa 0.370 and Exact Match accuracy of 45.22%, approaching Gemini-3.1-Pro with 30x fewer parameters and surpassing models up to 235B in scale. We further demonstrate that our judge produces reliable reward signals for downstream translation optimization, reducing the low-quality translation rate by over 20 percentage points under independent human evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。