构建药物监管差异检测基准,帮药企自动识别美欧审批要求分歧
RegDivergence-101: An LLM Benchmark for Cross-Jurisdiction Regulatory Contradiction Detection in Life Sciences
- 设计跨司法管辖区监管差异分类任务,区分一致、冲突与沉默三类关系
- 基于101对专家标注数据,大模型方法准确率达83%,显著优于传统方法
- 揭示监管沉默可被检测但需上下文理解,适合医药合规与AI交叉研究者
开发药物在美欧双地申报时的监管差异检测任务:给定一个FDA和一个EMA在同主题下的要求,判断其关系为一致(AGREE)、冲突(DIVERGE)或一方沉默(SILENT)。SILENT具有方向性(SILENT_FDA/EMA),分别记录并报告方向性F1。发布首个101对专家标注的基准数据集RegDivergence-101,标签基于三项同行评审研究及原始FDA/EMA/ICH指南文本;双标注者一致性Kappa达0.85。系统评估四种方法:词法启发式(宏F1 0.511,95% CI [0.411-0.605])、NLI交叉编码器(0.233)、义务级图RAG(0.663 [0.570-0.747])和扁平化大模型判别器(Claude Haiku,0.830 [0.747-0.908])。初步发现:沉默可语义识别但仅靠蕴含模型无法捕捉;义务图结构优于词法方法但逊于大模型上下文;大规模沉默检测应以语料级图构建为架构目标。本研究为任务定义与基线提供试点基础,第7节提出四个未覆盖监管领域及扩展路线。
原文摘要 · Abstract (English)
Pharmaceutical sponsors developing a drug for both the United States and the European Union must reconcile guidance issued independently by the FDA and the EMA. Where the two agencies require substantively the same thing, a sponsor can file once; where they diverge, a single trial design risks rejection in one region; where one agency is silent on a point the other regulates, the sponsor must infer obligations. Today this reconciliation is performed manually by regulatory-affairs experts. We introduce cross-jurisdiction regulatory divergence detection: given an FDA requirement and an EMA requirement on the same topic, classify their relationship as AGREE, DIVERGE, or SILENT. SILENT is inherently directional (SILENT_FDA vs. SILENT_EMA); we record direction per pair and report per-direction F1 alongside the collapsed label. We release RegDivergence-101, a 101-pair expert-grounded pilot evaluation benchmark (labels grounded in three peer-reviewed FDA/EMA comparison studies and primary FDA/EMA/ICH guidance text; dual-annotation inter-annotator kappa = 0.85), and systematically characterise a four-method baseline hierarchy: lexical heuristic (0.511 macro-F1, 95% CI [0.411-0.605]), NLI cross-encoder (0.233), obligation-level Graph-RAG (0.663 [0.570-0.747]), and flat LLM judge / Claude Haiku (0.830 [0.747-0.908]). Three directional observations emerge at pilot scale (n = 101): SILENT is semantically detectable but invisible to entailment-only formulations; pair-level obligation graphs improve over lexical methods but trail flat-LLM context (CIs partially overlapping); and corpus-level graph construction is the indicated architectural target for large-scale silent-detection. RegDivergence-101 is a pilot release establishing the task formulation and baseline hierarchy; four unrepresented regulatory domains and an expansion roadmap are described in Section 7.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。