比较三种投票策略，提升大模型钓鱼网址检测准确率
To Ensemble or Not: Assessing Majority Voting Strategies for Phishing Detection with Large Language Models
- 用多提示、多模型或混合方式集成大模型预测结果
- 当各模型表现接近时，集成效果优于单个模型
- 若模型差异大，最好直接选最强单个模型
大型语言模型（LLMs）的性能高度依赖于输入提示的质量。即使面对相同提示，不同训练过程也会导致输出结果不一致。为利用多个大模型的集体智慧并提升性能，本研究评估了三种用于文本分类的多数投票策略，聚焦于钓鱼网址检测任务。策略包括：(1) 基于提示的集成，即对同一模型使用不同提示生成响应后进行投票；(2) 基于模型的集成，即对单一提示调用多个模型并聚合结果；(3) 混合集成，将不同提示分配给多个模型后再聚合。分析表明，当各组件性能相近时，集成方法最有效；但若个体表现差距显著，集成效果未必超过最优单个模型或提示。此时应避免使用集成方法。
原文摘要 · Abstract (English)
The effectiveness of Large Language Models (LLMs) significantly relies on the quality of the prompts they receive. However, even when processing identical prompts, LLMs can yield varying outcomes due to differences in their training processes. To leverage the collective intelligence of multiple LLMs and enhance their performance, this study investigates three majority voting strategies for text classification, focusing on phishing URL detection. The strategies are: (1) a prompt-based ensemble, which utilizes majority voting across the responses generated by a single LLM to various prompts; (2) a model-based ensemble, which entails aggregating responses from multiple LLMs to a single prompt; and (3) a hybrid ensemble, which combines the two methods by sending different prompts to multiple LLMs and then aggregating their responses. Our analysis shows that ensemble strategies are most suited in cases where individual components exhibit equivalent performance levels. However, when there is a significant discrepancy in individual performance, the effectiveness of the ensemble method may not exceed that of the highest-performing single LLM or prompt. In such instances, opting for ensemble techniques is not recommended.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。