评测大模型在阿拉伯语复杂任务中的推理能力，发现少量示例可显著提升性能。
AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP
- 用少量上下文示例提升模型推理表现
- 三示例使情感分析准确率从35.3%升至87.5%
- 适合关注阿拉伯语NLP与模型推理的研究者
大型语言模型（LLMs）在推理能力和通用自然语言处理任务中表现出色，但在阿拉伯语数据上的表现仍缺乏深入研究，其丰富的词形变化、多样的方言和复杂的书写系统带来了挑战。本文对多个以推理为重点的LLM进行了全面基准测试，特别关注新推出的DeepSeek模型，在十五个阿拉伯语NLP任务上展开实验，采用零样本、少样本及微调等多种策略。结果表明：仅使用三个上下文示例即可在分类任务中平均提升13个F1点，如情感分析从35.3%提升至87.5%，句义相似度检测从56.1%升至87.0%；在零样本设置下，推理型DeepSeek架构比GPT-o4-mini基线平均高出12个F1点；基于LoRA的微调相较单纯扩大模型规模，额外带来最高8个F1点和BLEU值提升。代码已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown remarkable progress in reasoning abilities and general natural language processing (NLP) tasks, yet their performance on Arabic data, characterized by rich morphology, diverse dialects, and complex script, remains underexplored. This paper presents a comprehensive benchmarking study of multiple reasoning-focused LLMs, with a special emphasis on the newly introduced DeepSeek models, across a suite of fifteen Arabic NLP tasks. We experiment with various strategies, including zero-shot, few-shot, and fine-tuning. This allows us to systematically evaluate performance on datasets covering a range of applications to examine their capacity for linguistic reasoning under different levels of complexity. Our experiments reveal several key findings. First, carefully selecting just three in-context examples delivers an average uplift of over 13 F1 points on classification tasks-boosting sentiment analysis from 35.3% to 87.5% and paraphrase detection from 56.1% to 87.0%. Second, reasoning-focused DeepSeek architectures outperform a strong GPT o4-mini baseline by an average of 12 F1 points on complex inference tasks in the zero-shot setting. Third, LoRA-based fine-tuning yields up to an additional 8 points in F1 and BLEU compared to equivalent increases in model scale. The code is available at https://github.com/gufranSabri/deepseek-evals
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。