构建可追溯的孟加拉语农业问答基准,解决化学品推荐中的幻觉问题。
KrishokChat: A Provenance-Traceable Multi-Task Bengali Agricultural Benchmark with Safety-Critical Chemical Advisory

- 基于284份政府文献构建8.6万条带溯源标注的农业数据
- 模型在化学品推荐中仍存在4.05%~7.00%的幻觉率
- 专为农民真实语言和安全决策设计,适合农业AI安全研究
我们提出KrishokChat,一个包含85,979个实例的孟加拉语农业基准,源自284份政府出版物、13个机构及六个地区方言。该基准包含四个任务:通用知识问答、治疗方案问答、安全拒绝与重问、表格问答。另含1,000个独立田野访谈收集的真实农民查询,用于衡量对真实语言的迁移能力。每个实例均保留引用级溯源信息。治疗问答还提供剂量级可审计的化学成分追踪数组。评估了五个零样本基线和一个微调模型。闭卷知识无论模型规模如何均表现不足;使用理想证据可缩小差距,但化学品幻觉仍残留4.05%至7.00%。在通用问答任务上,基于KrishokChat微调的模型显著优于最强零样本基线。然而结构化表格推理与农民语言迁移仍难解决。探索性分析显示,微调会降低对安全关键问题的拒绝行为,表明需额外的安全对齐机制。我们公开发布KrishokChat,作为可追溯、可审计的资源,支持孟加拉语农业语言建模的可信与安全发展。
原文摘要 · Abstract (English)
We introduce KrishokChat, an 85,979-instance Bengali agricultural benchmark built from 284 government publications, 13 institutions, and six regional dialects. The benchmark comprises four tracks: General Knowledge QA, Treatment QA, Safety Refusal and Re-query, and Table QA. It also includes a 1,000-query Real-World Farmer Benchmark collected independently from field interviews to measure transfer to authentic farmer language. Every extracted instance retains provenance at the citation level. Treatment QA also provides a structured chemical-trace array for dosage-level auditability. We evaluated five zero-shot baselines and one fine-tuned model. Closed-book knowledge proves insufficient regardless of model scale. Oracle evidence narrows the gap, but leaves a persistent floor of 4.05 to 7.00% of chemical hallucinations. Fine-tuning on KrishokChat substantially outperforms the strongest zero-shot baseline on General QA Token F1. However, structured table reasoning and farmer-language transfer remain largely unsolved. An exploratory analysis further reveals that fine-tuning reduces refusal behavior on safety-critical queries, indicating that additional safety alignment is required beyond supervised fine-tuning. We release KrishokChat as a traceable benchmark and audit resource to support grounded and safety-aware agricultural language modeling for Bengali.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。