arXiv:2412.00765cs.CLcs.AI2024-12被引 11

让大模型自己生成测试题,自动评估其抗干扰能力。

SelfPrompt: Autonomously Evaluating LLM Robustness via Domain-Constrained Knowledge Guidelines and Refined Adversarial Prompts

  • 用领域知识图谱生成针对性强的对抗性提问
  • 自动生成并筛选高流畅度、高语义一致性的测试题
  • 无需外部数据集,适合特定场景下的模型评估

传统大语言模型鲁棒性评估方法依赖标准化基准,成本高且难以覆盖多样领域。本文提出一种自主评估框架,通过融合领域约束的知识图谱与优化后的对抗性提示,系统地从知识图谱三元组生成描述性句子,构建挑战性强的测试问题。这些由模型自身生成并经过严格过滤与优化的提示,确保文本流畅性和语义一致性,实现模型对自身鲁棒性的自我评估,无需依赖外部基准。我们在ChatGPT等专有模型及Llama-3.1、Phi-3、Mistral等开源模型上进行了广泛测试,结果表明该方法显著降低对传统数据的依赖,提供高效且聚焦于特定领域的鲁棒性评估手段。

原文摘要 · Abstract (English)

Traditional methods for evaluating the robustness of large language models (LLMs) often rely on standardized benchmarks, which can escalate costs and limit evaluations across varied domains. This paper introduces a novel framework designed to autonomously evaluate the robustness of LLMs by incorporating refined adversarial prompts and domain-constrained knowledge guidelines in the form of knowledge graphs. Our method systematically generates descriptive sentences from domain-constrained knowledge graph triplets to formulate adversarial prompts, enhancing the relevance and challenge of the evaluation. These prompts, generated by the LLM itself and tailored to evaluate its own robustness, undergo a rigorous filtering and refinement process, ensuring that only those with high textual fluency and semantic fidelity are used. This self-evaluation mechanism allows the LLM to evaluate its robustness without the need for external benchmarks. We assess the effectiveness of our framework through extensive testing on both proprietary models like ChatGPT and open-source models such as Llama-3.1, Phi-3, and Mistral. Results confirm that our approach not only reduces dependency on conventional data but also provides a targeted and efficient means of evaluating LLM robustness in constrained domains.

大模型评估对抗测试自评估知识图谱

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。