评测8个开源大模型识别人类反犹主义内容能力，发现新提示法显著提升效果与可解释性。
Evaluating Large Language Models for Detecting Antisemitism
- 设计新型引导式思维链提示（Guided-CoT），融入领域知识提升识别性能。
- Llama 3.1 70B在无微调情况下超越微调版GPT-3.5，错误率降低32%。
- 提出语义偏差度量指标，揭示模型解释逻辑中的矛盾与不一致现象。
检测仇恨内容是重要且具有挑战性的问题。自动化工具如机器学习模型可提供帮助，但需持续训练以适应社交媒体的快速变化。本文评估了八个开源大语言模型（LLMs）在识别反犹主义内容方面的能力，特别采用上下文定义策略。我们研究了不同提示技巧，并设计了一种类似思维链的新提示方法——引导式思维链（Guided-CoT）。结果显示，注入领域特定思考能提升性能和实用性。Guided-CoT有效处理上下文政策，减少所有模型的拒绝响应，无论解码配置、模型规模或推理能力如何。值得注意的是，未微调的Llama 3.1 70B表现优于微调后的GPT-3.5。此外，我们分析了模型错误并引入度量指标来量化模型生成解释中的语义偏差，揭示出各模型在实用性、可解释性和可靠性上的显著差异。代码与资源见：https://github.com/idramalab/quantify-llm-explanations
原文摘要 · Abstract (English)
Detecting hateful content is a challenging and important problem. Automated tools, like machine-learning models, can help, but they require continuous training to adapt to the ever-changing landscape of social media. In this work, we evaluate eight open-source LLMs' capability to detect antisemitic content, specifically leveraging in-context definition. We also study how LLMs understand and explain their decisions given a moderation policy as a guideline. First, we explore various prompting techniques and design a new CoT-like prompt, Guided-CoT, and find that injecting domain-specific thoughts increases performance and utility. Guided-CoT handles the in-context policy well, improving performance and utility by reducing refusals across all evaluated models, regardless of decoding configuration, model size, or reasoning capability. Notably, Llama 3.1 70B outperforms fine-tuned GPT-3.5. Additionally, we examine LLM errors and introduce metrics to quantify semantic divergence in model-generated rationales, revealing notable differences and paradoxical behaviors among LLMs. Our experiments highlight the differences observed across LLMs' utility, explainability, and reliability. Code and resources available at: https://github.com/idramalab/quantify-llm-explanations
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。