用共识机制从噪声数据中挑出可靠样本，低成本训练更准的告警过滤模型。
ConceptRM: The Quest to Mitigate Alert Fatigue through Consensus-Based Purity-Driven Data Cleaning for Reflection Modelling
- 通过扰动数据+多模型协同学习，基于共识识别可靠负样本
- 仅少量专家标注下，误报拦截率比顶尖大模型高53.31%（域内）
- 适合需要降低告警疲劳、资源有限的智能系统部署场景
在涉及智能体的应用中，大量由智能体生成的告警（多数为误报）可能导致用户产生告警疲劳，从而忽视关键问题。常见做法是利用用户验证反馈构建标注数据，训练一个反思模型作为过滤器来拦截误报。然而，这类数据常因在生产环境中采集而具有噪声。手动标注清洗成本过高，本文提出一种新方法 ConceptRM，仅需少量专家标注作为锚点，通过生成不同噪声比例的扰动数据集，并使用协同教学训练多个独立模型进行协作学习。通过分析这些模型的共识决策，有效从噪声数据中识别出可靠的负样本。实验表明，ConceptRM 在极低标注成本下显著提升了误报拦截能力，在域内数据集上优于多个前沿大模型达53.31%，在域外数据集上提升41.67%。
原文摘要 · Abstract (English)
In many applications involving intelligent agents, the overwhelming volume of alerts (mostly false) generated by the agents may desensitize users and cause them to overlook critical issues, leading to the so-called ''alert fatigue''. A common strategy is to train a reflection model as a filter to intercept false alerts with labelled data collected from user verification feedback. However, a key challenge is the noisy nature of such data as it is often collected in production environments. As cleaning noise via manual annotation incurs high costs, this paper proposes a novel method ConceptRM for constructing a high-quality corpus to train a reflection model capable of effectively intercepting false alerts. With only a small amount of expert annotations as anchors, ConceptRM creates perturbed datasets with varying noise ratios and utilizes co-teaching to train multiple distinct models for collaborative learning. By analyzing the consensus decisions of these models, it effectively identifies reliable negative samples from a noisy dataset. Experimental results demonstrate that ConceptRM significantly enhances the interception of false alerts with minimal annotation cost, outperforming several state-of-the-art LLM baselines by up to 53.31% on in-domain datasets and 41.67% on out-of-domain datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。