超小模型也能精准检测机器翻译关键错误，适合本地部署。
How Small Can You Go? Compact Language Models for On-Device Critical Error Detection in Machine Translation
- 用轻量校准+投票机制，让小模型实现高精度错误检测。
- 10亿参数的Gemma-3-1B在合成数据上达F1-ERR=0.98，延迟仅400ms。
- 适合对隐私和成本敏感的实时翻译场景，代码数据开源。
大语言模型在机器翻译评估中表现优异，但规模与成本限制了其在边缘设备和隐私敏感场景的应用。本文探讨在保持关键错误检测能力的前提下，模型能小到什么程度。聚焦英德翻译中的关键错误检测（CED），评测了参数量小于20亿的模型（LFM2-350M、Qwen-3-0.6B/1.7B、Llama-3.2-1B-Instruct、Gemma-3-1B），覆盖WMT21、WMT22和SynCED-EnDe-2025数据集。采用标准化提示、轻量级logit偏置校准与多数投票策略，同时报告语义质量（MCC、F1-ERR/F1-NOT）与计算开销（VRAM、延迟、吞吐）。结果表明，约10亿参数模型为最佳平衡点：经合并权重微调后，Gemma-3-1B在SynCED-EnDe-2025上实现MCC=0.77、F1-ERR=0.98，单样本延迟400ms（MacBook Pro M4 Pro，24GB内存）。更大模型（Qwen-3-1.7B）虽取得更高绝对MCC（+0.11），但开销更高；超小模型（0.6B）经少样本校准仍可用，但对实体与数字错误检测不足。结论：经轻量校准与小样本监督增强的紧凑指令微调模型，可实现可信、本地化的机翻关键错误检测，适用于真实世界翻译流水线中的私密、低成本筛查。所有数据集、提示与脚本均开源于GitHub。
原文摘要 · Abstract (English)
Large Language Models (LLMs) excel at evaluating machine translation (MT), but their scale and cost hinder deployment on edge devices and in privacy-sensitive workflows. We ask: how small can you get while still detecting meaning-altering translation errors? Focusing on English->German Critical Error Detection (CED), we benchmark sub-2B models (LFM2-350M, Qwen-3-0.6B/1.7B, Llama-3.2-1B-Instruct, Gemma-3-1B) across WMT21, WMT22, and SynCED-EnDe-2025. Our framework standardizes prompts, applies lightweight logit-bias calibration and majority voting, and reports both semantic quality (MCC, F1-ERR/F1-NOT) and compute metrics (VRAM, latency, throughput). Results reveal a clear sweet spot around one billion parameters: Gemma-3-1B provides the best quality-efficiency trade-off, reaching MCC=0.77 with F1-ERR=0.98 on SynCED-EnDe-2025 after merged-weights fine-tuning, while maintaining 400 ms single-sample latency on a MacBook Pro M4 Pro (24 GB). At larger scale, Qwen-3-1.7B attains the highest absolute MCC (+0.11 over Gemma) but with higher compute cost. In contrast, ultra-small models (0.6B) remain usable with few-shot calibration yet under-detect entity and number errors. Overall, compact, instruction-tuned LLMs augmented with lightweight calibration and small-sample supervision can deliver trustworthy, on-device CED for MT, enabling private, low-cost error screening in real-world translation pipelines. All datasets, prompts, and scripts are publicly available at our GitHub repository.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。