在少样本社会科学研究中，上下文学习比指令微调更有效。
Instruction Tuning Vs. In-Context Learning: Revisiting Large Language Models in Few-Shot Computational Social Science
- 用上下文学习直接从示例中快速适应任务，无需梯度更新。
- 增加样本数量但不考虑质量，反而可能降低模型表现。
- 上下文学习在提示策略中优于零样本和思维链方法。
大型语言模型（LLM）在计算社会科学（CSS）任务中的实际应用主要依赖于指令微调（IT）或上下文学习（ICL）。尽管指令微调在多个任务上表现出色，上下文学习则通过示例实现无需显式梯度更新的任务适配，速度更快。本文评估了在少样本CSS任务中，使用指令微调与上下文学习的分类性能。实验结果表明，上下文学习在大多数任务中均优于指令微调。此外，我们研究了训练样本数量增加对模型性能的影响，发现仅增加样本数量而忽视其质量，并不能持续提升性能，甚至可能导致性能下降。最后，我们比较了三种提示策略，证明上下文学习优于零样本和思维链（CoT）策略。研究强调了上下文学习在少样本设置下处理社会科学研究任务的优势，并指出优化样本质量和提示策略对提升分类性能的重要性。代码将公开。
原文摘要 · Abstract (English)
Real-world applications of large language models (LLMs) in computational social science (CSS) tasks primarily depend on the effectiveness of instruction tuning (IT) or in-context learning (ICL). While IT has shown highly effective at fine-tuning LLMs for various tasks, ICL offers a rapid alternative for task adaptation by learning from examples without explicit gradient updates. In this paper, we evaluate the classification performance of LLMs using IT versus ICL in few-shot CSS tasks. The experimental results indicate that ICL consistently outperforms IT in most CSS tasks. Additionally, we investigate the relationship between the increasing number of training samples and LLM performance. Our findings show that simply increasing the number of samples without considering their quality does not consistently enhance the performance of LLMs with either ICL or IT and can sometimes even result in a performance decline. Finally, we compare three prompting strategies, demonstrating that ICL is more effective than zero-shot and Chain-of-Thought (CoT). Our research highlights the significant advantages of ICL in handling CSS tasks in few-shot settings and emphasizes the importance of optimizing sample quality and prompting strategies to improve LLM classification performance. The code will be made available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。