发现古文字翻译数据污染，导致模型性能虚高，提出真实评估基准。
Data Contamination in Neural Hieroglyphic Translation: A Reproducibility Study
- 检测到2%测试样本与训练集完全重复，造成评分严重失真。
- 去重后模型最高仅39.2 BLEU，远低于原报告的61.5。
- 为濒危文字翻译提供可复现的纯净测试集和真实性能基准。
古代及濒危语言的NLP面临独特挑战：数据稀缺、难以扩展且源于程式化语料，数据质量问题尤为关键却极少被审计。为理解当前神经机器翻译对这类语言的真实能力，我们研究象形文字到德语的翻译任务，此前研究声称使用微调的M2M-100模型达到61.5 BLEU。我们复现后仅得37.0 BLEU。分析发现，测试集中2%（16/50）的目标文本在训练中完全重复出现（8-gram重叠达70%阈值时占比50%）。这些污染样本得分高达83.8 BLEU / 0.924 COMET-22，而干净样本仅为30.9–39.2 BLEU / 0.622–0.676 COMET-22，涵盖两种架构的五种模型配置。文档级去重仅降低4.6点，因8个污染样本仍通过其他来源残留；必须进行目标级去重。我们发布一个34样本的去污染测试集，并建立修正后的基线（30.9–39.2 BLEU），为该濒危书写系统提供真实性能评估。
原文摘要 · Abstract (English)
Ancient and endangered languages pose a unique challenge for NLP: their datasets are inherently scarce, difficult to expand, and built from formulaic corpora -- making data-quality issues especially consequential yet rarely audited. Motivated by the need to understand what current NMT can realistically achieve for such languages, we investigate hieroglyphic-to-German translation, where a recent study reported 61.5 BLEU using fine-tuned M2M-100. Our reproduction yields only 37.0 BLEU with the released model. Investigating this gap, we find 2\% of test targets appear identically in training (16/50; 50\% under 8-gram overlap at 70\% threshold). This contamination inflates scores dramatically: contaminated samples achieve up to 83.8 BLEU / 0.924 COMET-22 versus 30.9--39.2 BLEU / 0.622--0.676 COMET-22 on clean samples across five model configurations spanning two architectures. Document-level decontamination reduces contaminated BLEU by only 4.6 points because 8/16 targets persist via other source documents -- target-level deduplication is required. We release a decontaminated 34-sample test set and establish corrected baselines (30.9--39.2 BLEU), providing a realistic assessment of NMT capability for this endangered writing system.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。