用双阶段对比学习提升尼泊尔语梗图的仇恨言论与情感识别准确率。
ZeroR@CHiPSAL 2026: Two-Stage Vision-Language Adaptation with Contrastive Learning for Nepali Meme Classification
- 分两阶段训练：先微调生成分类头，再用对比损失优化视觉语言主干。
- 在仇恨言论任务中达F1 0.797，情感分析任务中达F1 0.518，位列第二和第四。
- 直接理解天城文，避免翻译误差，适合低资源南亚语言研究者参考。
本文针对CHiPSAL 2026多模态仇恨言论与情感检测共享任务，提出一种基于Qwen3-VL-8B-Instruct的双阶段视觉语言适应系统，支持原生天城文处理。该方法在尼泊尔语梗图上完成二分类仇恨言论检测与三分类情感分析。采用两阶段训练流程：（1）使用LoRA微调并搭配MLP投影头进行生成式分类；（2）通过监督InfoNCE损失对模型主干进行对比学习微调。针对类别不平衡问题，引入少数类过采样、图像增强与焦点损失。推理阶段融合阶段一的词元概率与阶段二的分类分数，权重经验证集调优。系统摒弃了独立的OCR与翻译流水线，实现端到端处理。最终在仇恨言论检测中取得F1 0.797（第二名），情感分析中取得F1 0.518（第四名）。论文提供详尽消融实验、错误分析，并深入探讨大模型在低资源南亚语言上的适配策略。
原文摘要 · Abstract (English)
This paper presents our system for the CHiPSAL 2026 shared task on multimodal hate speech and sentiment detection in Nepali memes. We address both subtasks: binary hate speech classification and three-class sentiment analysis. Our approach adapts the Robust Adaptation of Hateful Meme Detection (RA-HMD) framework using Qwen3-VL-8B-Instruct, a state-of-the-art vision-language model with native Devanagari support. We employ a two-stage training pipeline: (1) LoRA fine-tuning with an MLP projection head for generative classification, and (2) contrastive backbone fine-tuning with supervised InfoNCE loss. We handle class imbalance through minority oversampling, image augmentation, and focal loss. At inference, we ensemble Stage 1 token probabilities with Stage 2 classifier scores using validation-tuned weights. Our end-to-end approach eliminates error propagation from separate OCR and translation pipelines by leveraging the model's native Devanagari understanding. Our system achieved \textbf{2nd place} on hate speech detection (F1: 0.797) and \textbf{4th place} on sentiment analysis (F1: 0.518). We provide detailed ablations, error analysis, and insights into adapting large vision-language models for low-resource South Asian languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。