自动优化提示词与动态选例,提升政治科学文本分类准确率
Enhancing LLM-Based Text Classification in Political Science: Automatic Prompt Optimization and Dynamic Exemplar Selection for Few-Shot Learning
- 用任务样本自动优化提示词,避免人工试错
- 动态挑选最相关示例,提升每条输入的上下文适配性
- 模拟多人编码共识,低成本保证结果可靠
大型语言模型(LLMs)在政治科学文本分类中潜力巨大,但其效果常依赖高质量提示词和示例。为此,我们提出一个三阶段框架:通过任务特定示例自动优化提示词,消除主观试错,生成符合人类标准的结构化提示;第二阶段动态选择最相关示例,为每个查询提供上下文恰当的引导;第三阶段采用共识机制,模拟多位人工编码员对单一任务的判断,融合多个LLM输出以实现高可靠性与一致性,同时显著降低成本。该方法在情感分析、立场识别和竞选广告语气分类等任务上均提升了分类准确率,无需任务特定模型微调或大量手动调整提示词。框架不仅提高准确性、可解释性与透明度,还为政治科学应用提供了低成本、可扩展的解决方案。开源Python工具包 PoliPrompt 已发布于GitHub。
原文摘要 · Abstract (English)
Large language models (LLMs) offer substantial promise for text classification in political science, yet their effectiveness often depends on high-quality prompts and exemplars. To address this, we introduce a three-stage framework that enhances LLM performance through automatic prompt optimization, dynamic exemplar selection, and a consensus mechanism. Our approach automates prompt refinement using task-specific exemplars, eliminating speculative trial-and-error adjustments and producing structured prompts aligned with human-defined criteria. In the second stage, we dynamically select the most relevant exemplars, ensuring contextually appropriate guidance for each query. Finally, our consensus mechanism mimics the role of multiple human coders for a single task, combining outputs from LLMs to achieve high reliability and consistency at a reduced cost. Evaluated across tasks including sentiment analysis, stance detection, and campaign ad tone classification, our method enhances classification accuracy without requiring task-specific model retraining or extensive manual adjustments to prompts. This framework not only boosts accuracy, interpretability and transparency but also provides a cost-effective, scalable solution tailored to political science applications. An open-source Python package (PoliPrompt) is available on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。