用希腊语训练机器人政策,发现评估方法比翻译更难。
Measuring Language Transfer in Robot Policies: Adding Greek to a Cosmos3 Vision-Language-Action Policy

- 仅用机器重述指令,不改架构添加希腊语
- 双语训练比单语高6.7-7.1分,达英语性能的40%
- 需多轮种子验证,避免评估误导
机器人基础模型主要在英语环境下训练与评估,多数语言缺乏机器人示范数据集。本文研究在开放的视觉-语言-动作系统中仅通过机器重述指令加入希腊语,不修改模型结构。核心挑战在于评估而非翻译。多种常见评估指标存在缺陷:颜色直方图奖励噪声,单任务基准在正确希腊语下得84.6%,错误指令下仍达82.6%;训练损失无法预测希腊语表现;单次运行结果受随机种子主导。在包含90项任务的判别性测试套件中,三组种子每组三轮,无希腊语示范的多语言文本塔始终处于错误指令最低水平,而纯希腊语训练最多仅超出对照组2.7分。双语训练稳定领先对照组6.7-7.1分,达到英语性能约五分之二。政策还对翻译表述产生过拟合;每任务使用七种表述训练可使该惩罚减半。从语言适配的世界模型热启动并解冻文本塔反而降低性能。结果表明,低资源语言机器人政策本地化需满足两项实践要求:建立可信零基准以信任评估指标,跨种子重复验证低资源语言结果。
原文摘要 · Abstract (English)
Robot foundation models are trained and evaluated predominantly in English, and robot demonstration corpora do not exist for most languages. We study the addition of Greek to an open vision-language-action stack using only machine-rephrased instructions and no architecture changes. The main challenge is measurement rather than translation. Several plausible instruments produce false conclusions: a color-histogram metric rewards noise, a single-goal benchmark scores 84.6% under correct Greek and 82.6% under deliberately wrong instructions, training loss fails to predict Greek success, and single-run comparisons are dominated by seed variation. On a discriminative ninety-task suite with three seeds per arm, a multilingual text tower without Greek demonstrations remains at its wrong-instruction floor, while Greek-only training exceeds its control by at most 2.7 points. Bilingual training yields a consistent 6.7-7.1 point margin over its control and reaches about two fifths of English performance. The policy also overfits the translator's phrasing; training on seven phrasings per task approximately halves this penalty. Warm-starting from a language-adapted world model and unfreezing the text tower both degrade performance. The results support two practical requirements for low-resource robot-policy localization: build a guaranteed null before trusting a metric, and replicate low-resource-language results across seeds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。