首次揭示去中心化GRPO的攻击漏洞并提出防御方案
Hail to the Thief: Exploring Attacks and Defenses in Decentralised GRPO
- 设计多种恶意节点污染模型的攻击方法
- 50轮内实现数学与编码任务100%攻击成功率
- 提出概率校验与LLM判别两种防御机制
Group Relative Policy Optimization (GRPO) 在大语言模型后训练中广泛应用。在GRPO中，通过强化学习从模型对提示的回应中学习偏好行为。由于通信量小，GRPO天然适合去中心化训练，多个节点可并行回答提示，仅需交换字符串形式的生成结果。本文首次探索去中心化GRPO的鲁棒性，提出多种对抗攻击，并设计相应防御措施。恶意节点通过共享被污染的生成结果来毒化良性模型。实验表明，在数学和编程任务上，攻击者可在50次迭代内实现100%成功率。为此，我们提出两种防御机制：一种基于检查生成结果的逻辑概率，另一种利用大型语言模型作为判别器过滤输出。这两种方法可有效抵御多数攻击，仅无法防范导致冗长但概念正确的拒绝服务（DoS）攻击。相关代码已公开于：https://github.com/gensyn-ai/HTTT。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) has demonstrated wide adoption in the post-training of Large Language Models (LLMs). In GRPO, prompts are answered by the model and preferred behaviour is learnt via reinforcement learning. Owing to the small communication volume, GRPO is inherently suitable for decentralised training as the prompts can be concurrently answered by multiple nodes and these completions are exchanged in the form of strings. In this work, we explore the robustness of decentralised GRPO by presenting the first adversarial attacks and countermeasures. We present a diverse set of attacks where malicious nodes poison benign models by sharing their poisoned completions. We demonstrate these attacks on math and coding tasks and show that an adversary can achieve attack success rates of up to 100% in as few as 50 iterations. Moreover, to mitigate the attacks, we propose two defense mechanisms that check logit probabilities of completions or utilize an LLM judge to filter completions. The defenses prevent all but the DoS attack that causes unnecessarily lengthy but conceptually correct completions. The code of both attacks and defenses can be found at: https://github.com/gensyn-ai/HTTT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。