让大模型自己总结分类原则，比人工写规则更有效。
Don't Just Demo, Teach Me the Principles: A Principle-Based Multi-Agent Prompting Strategy for Text Classification
- 多个AI代理分析样例，共同提炼出分类原则
- 在多个数据集上比零样本提示提升1.55%~19.37%的宏平均F1
- 生成原则质量高，适合希望低成本提升分类性能的研究者
我们提出PRINCIPLE-BASED PROMPTING，一种用于文本分类的多代理提示策略。该方法首先让多个大语言模型（LLM）代理独立基于有或无标签的示范样本生成候选分类原则，再通过一个终合代理将这些原则整合为最终版本，最后由分类代理执行下游分类任务。在不同规模大模型上的二分类和多分类数据集上进行的大量实验表明，该方法不仅在宏平均F1得分上比零样本提示提升1.55%至19.37%，还优于其他强基线（如CoT和stepback提示）。在两个私有数据集上，该方法生成的原则使大模型表现优于人工设计的原则。此外，该多代理方法与基于示范的少样本提示相当甚至更优，但推理成本显著更低。消融实验表明，标签信息和多代理协作框架对生成高质量原则至关重要。
原文摘要 · Abstract (English)
We present PRINCIPLE-BASED PROMPTING, a simple but effective multi-agent prompting strategy for text classification. It first asks multiple LLM agents to independently generate candidate principles based on analysis of demonstration samples with or without labels, consolidates them into final principles via a finalizer agent, and then sends them to a classifier agent to perform downstream classification tasks. Extensive experiments on binary and multi-class classification datasets with different sizes of LLMs show that our approach not only achieves substantial performance gains (1.55% - 19.37%) over zero-shot prompting on macro-F1 score but also outperforms other strong baselines (CoT and stepback prompting). Principles generated by our approach help LLMs perform better on classification tasks than human crafted principles on two private datasets. Our multi-agent PRINCIPLE-BASED PROMPTING approach also shows on-par or better performance compared to demonstration-based few-shot prompting approaches, yet with substantially lower inference costs. Ablation studies show that label information and the multi-agent cooperative LLM framework play an important role in generating high-quality principles to facilitate downstream classification tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。