用新方法定位大模型提示中的模糊位置,提升故障诊断精度。
Localizing Prompt Ambiguity in Large Language Models with Probe-Targeted Attribution

- 通过训练探测器识别模糊与清晰提示,将得分反向归因到词元表示。
- 在合成数据集上达0.840 AUROC,金标准上达0.891 AUROC。
- 适用于代码、数学、写作场景,且跨领域仍有效。
提示模糊是大语言模型常见失败原因,但难以定位,因其为提示的潜在属性,而现有归因方法仅针对可观察输出(如logits或生成词元)。我们提出PRIG,一种利用探测器logit进行梯度归因的方法,将隐式模糊性归因于词元位置。具体而言,PRIG训练线性探测器以区分清晰与模糊提示,并将探测器得分归因于残差流中的早期词元表示。为实现词元级评估,我们在编码、数学和写作任务中通过重写每条提示的关键句子构建合成模糊数据集,并补充人工标注的金标准基准。在此设定下,PRIG在综合合成基准上取得0.840 AUROC,金标准上达0.891 AUROC,显著优于基线梯度归因方法;其在句级模糊识别上亦超越GPT-5.4,且在域外任务中仍保留有效信号。结果表明,PRIG是定位提示模糊部分的实用工具,更广泛地说明:可通过中间表示而非输出层面归因来定位提示的潜在属性。
原文摘要 · Abstract (English)
Prompt ambiguity is a common source of failure in large language models, but is difficult to localize because it is a latent property of the prompt, while existing attribution methods are designed to explain observable outputs such as logits or generated tokens. We introduce PRIG, a gradient attribution method that uses a probe logit to attribute latent ambiguity to token positions. Specifically, PRIG trains a linear probe to distinguish clear prompts from ambiguous prompts and attributes the probe score to earlier token representations in the residual stream. To enable token-level evaluation, we construct synthetic ambiguity datasets across coding, math, and writing by rewriting one task-critical sentence per prompt, and complement them with a human-written gold benchmark. In this setting, PRIG localizes ambiguous spans substantially better than gradient attribution baselines, achieving 0.840 AUROC on the combined synthetic benchmark and 0.891 AUROC on the gold set. It also outperforms GPT-5.4 on sentence-level ambiguity identification and retains useful signal out-of-domain. These results establish PRIG as a practical tool for identifying which parts of a prompt are ambiguous. More broadly, they suggest that latent prompt properties can be localized through intermediate representations, rather than through output-level attribution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。