arXiv:2510.17535cs.IR2025-10被引 4

角色扮演能显著提升大模型零样本排序效果,研究揭示其内在机制。

How role-play shapes relevance judgment in zero-shot LLM rankers

  • 通过因果干预分析角色设定如何影响排序判断
  • 角色信息主要在早期层编码,与任务指令交互
  • 适合关注提示工程与模型可解释性的研究人员

大语言模型作为零样本排序器展现出巨大潜力,但其性能高度依赖提示设计。特别是角色扮演类提示(赋予模型特定身份)常带来更鲁棒、准确的排序结果。然而,角色扮演的作用机制与多样性仍不清晰,限制了有效应用与可解释性。本文系统研究角色扮演变体对零样本大模型排序器的影响。采用机制可解释性中的因果干预技术,追踪角色信息如何塑造相关性判断。结果表明:(1) 精心设计的角色描述对排序质量有显著影响;(2) 角色信息主要在早期层编码,并在中层与任务指令交互,与查询或文档表征互动有限。我们识别出一组对角色条件相关性判断至关重要的注意力头。这些发现不仅揭示了角色扮演在大模型排序中的内在机制,也为信息检索及其他零样本场景下的提示设计提供了指导,拓展了角色扮演的应用前景。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have emerged as promising zero-shot rankers, but their performance is highly sensitive to prompt formulation. In particular, role-play prompts, where the model is assigned a functional role or identity, often give more robust and accurate relevance rankings. However, the mechanisms and diversity of role-play effects remain underexplored, limiting both effective use and interpretability. In this work, we systematically examine how role-play variations influence zero-shot LLM rankers. We employ causal intervention techniques from mechanistic interpretability to trace how role-play information shapes relevance judgments in LLMs. Our analysis reveals that (1) careful formulation of role descriptions have a large effect on the ranking quality of the LLM; (2) role-play signals are predominantly encoded in early layers and communicate with task instructions in middle layers, while receiving limited interaction with query or document representations. Specifically, we identify a group of attention heads that encode information critical for role-conditioned relevance. These findings not only shed light on the inner workings of role-play in LLM ranking but also offer guidance for designing more effective prompts in IR and beyond, pointing toward broader opportunities for leveraging role-play in zero-shot applications.

大模型排序提示工程可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。