arXiv:2607.15555cs.IR2026-07

发现大模型内部的语义相关性随层数增强,且跨语言可部分迁移。

LLMs Encode Relevance as a Layer-Wise Cross-Lingual Signal

论文配图:LLMs Encode Relevance as a Layer-Wise Cross-Lingual Signal
图 1 · 摘自论文原文
  • 通过逐层探测激活值,发现相关性信号在中后层更易提取。
  • 内部信号比生成判断更准确,能更好保持系统排名顺序。
  • 跨语言传递能力有限,但揭示了模型内表示差异的机制。

大型语言模型(LLMs)在信息检索中常被用作相关性判断与重排序工具,但现有研究多聚焦输出结果,缺乏对模型内部相关性表征的理解。本文以TREC DL20和MIRACL数据集为基础,使用类似UMBRELA的提示引导4-9B参数的指令微调模型,提取每一层Transformer的最后令牌激活值,并训练线性探测器预测相关性标签。结果表明,查询-文档相关性以深度依赖的方式编码:早期层探测性能弱,中后层最强,说明相关性在上下文整合后更易线性解码。更重要的是,经验证选择的探测器性能可匹配甚至超越生成判断,且更准确保留系统排名,揭示了内部表征与外部表达之间的分离。多语言实验显示信号部分跨语言可迁移,但效果仍弱于同语言解码。本工作从表示层面提供了对基于LLM的相关性评估的新视角,有助于诊断相关性何时浮现、生成判断失准时的内部证据,以及不同语言、数据集与模型家族间的表示差异。

原文摘要 · Abstract (English)

Large language models (LLMs) are increasingly used in information retrieval (IR) pipelines as relevance judges and re-rankers. Yet most analyses remain output-centric, evaluating generated labels or scores while offering limited insight into how relevance is represented inside the model. In this work, we study whether query-document (q-d) relevance is linearly decodable from residual-stream activations in instruction-tuned LLMs, how this signal compares with generated relevance judgments, and whether it transfers across languages. Using the TREC DL20 and MIRACL evaluation collections, we guide medium-scale LLMs (4-9B parameters) with UMBRELA-style relevance judgment prompts, extract last-token activations from every transformer layer, and train linear probes to predict relevance labels. We compare probe predictions with generated judgments and use TREC DL20 to test whether probe-derived pseudo-labels preserve system rankings against human judgments. Our results suggest that q-d relevance is encoded as a depth-dependent signal: probe performance is weak in early layers and strongest in middle-to-late layers, indicating that relevance becomes more linearly accessible after contextual integration. Most importantly, in several models, validation-selected probes match or outperform generated judgments and better preserve system rankings, revealing a separation between internal relevance representation and external expression. Multilingual experiments suggest partial cross-language portability, although transfer remains weaker than within-language decoding. Overall, this work provides a representation-level perspective on LLM-based relevance assessment. Layer-wise probing can help diagnose where relevance emerges, when generated judgments fail to reflect internally available evidence, and how relevance representations vary across languages, datasets, and model families.

大模型相关性表征跨语言迁移层间探测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。