用大模型链式思考生成黑盒检索模型的对抗样本。
Attack-in-the-Chain: Bootstrapping Large Language Models for Attacks Against Black-box Neural Ranking Models
- 基于链式思考构建大模型与检索模型的交互路径。
- 在两个搜索数据集上成功生成有效对抗样本，攻击成功率高。
- 适合研究模型鲁棒性或对抗攻击的从业者参考。
神经排序模型（NRMs）在检索性能方面表现出色，但对攻击的敏感性也高于以往模型。为揭示并应对这一脆弱性，本文提出一种名为 Attack-in-the-Chain 的新型排序攻击框架，通过链式思考（CoT）提示追踪大语言模型（LLMs）与 NRMs 的交互，实现在黑盒设置下生成对抗样本。方法首先将排名高于目标文档的锚文档作为推理链中的节点，动态分配每个节点的扰动词数，并引导 LLM 执行攻击。随后在每一步推理中验证各节点的攻击效果，进而生成下一步推理。在两个网页搜索基准测试上的实验表明，该方法具有显著有效性。
原文摘要 · Abstract (English)
Neural ranking models (NRMs) have been shown to be highly effective in terms of retrieval performance. Unfortunately, they have also displayed a higher degree of sensitivity to attacks than previous generation models. To help expose and address this lack of robustness, we introduce a novel ranking attack framework named Attack-in-the-Chain, which tracks interactions between large language models (LLMs) and NRMs based on chain-of-thought (CoT) prompting to generate adversarial examples under black-box settings. Our approach starts by identifying anchor documents with higher ranking positions than the target document as nodes in the reasoning chain. We then dynamically assign the number of perturbation words to each node and prompt LLMs to execute attacks. Finally, we verify the attack performance of all nodes at each reasoning step and proceed to generate the next reasoning step. Empirical results on two web search benchmarks show the effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。