低资源语言图文检索性能差,根源在编码器前层语言轨迹发散。
Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?
- 通过替换语言末尾隐藏状态,提升低资源语言表现
- 替换后斯瓦希里语检索准确率从22.1%升至69.1%
- 方法可复现且不损害高资源语言性能
当前多语言图文编码器虽覆盖数百种语言,但在低资源语言(如斯瓦希里语)上的图像-文本检索性能仍比高资源语言(如英语)低30个百分点以上。本文探究该差距的根源,发现线性语言方向并非主因,而是编码器前向路径中各语言的序列结束(EOS)隐藏状态轨迹随深度发散所致。将斯瓦希里语的EOS状态替换为三步前的英文对应值,可使检索准确率从22.1%大幅提升至69.1%;三组对照实验排除了位置聚合和英语特异性等干扰因素。此外,在训练阶段引入前层统一投影机制,使低资源语言在XM3600数据集(1,000张子集)上提升9.6 / 17.1个百分点,跨三个基准测试均有效且保持高资源语言性能。
原文摘要 · Abstract (English)
Recent multilingual vision--language encoders cover hundreds of languages in a single model, yet on two state-of-the-art instances retrieval on low-resource languages (LRL; e.g. Swahili) trails high-resource ones (HRL; e.g. English) by $30^+$\,pp. We ask where in the trained encoder this gap is located. Prior modality-gap and cross-lingual subspace work suggests a linear language direction at the output crowds out alignment-relevant geometry. We falsify this: LEACE drives the linear language classifier from $>99\%$ to near chance and iterated INLP to $37$--$50\%$ while LRL retrieval moves within $\pm 1.5$\,pp and all tier means within $2.2$\,pp, tracking random controls. The linear bias is a \emph{symptom}, not the cause. Instead, the alignment-causal factor lies along the encoder's forward path: the EOS (end-of-sequence) hidden state's per-language trajectory diverges with depth. Substituting the EOS with its parallel English value three blocks before the projector lifts Swahili from $22.1\%$ to $69.1\%$ on one encoder (and reproduces on the other); three controls rule out pooled-position tautology and English specificity. A front-layer trunk that pulls each language's projection toward the parallel-content centroid corroborates the diagnosis at training time, recovering $+9.6$ / $+17.1$\,pp on LRL XM3600 retrieval (1{,}000-image subset), with consistent gains across three further benchmarks while preserving HRL performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。