用检索增强分类技术,安全高效地识别机密文档。
Retrieval Augmented Classification for Confidential Documents
- 通过外部向量库检索匹配,避免模型参数泄露敏感信息
- 在不平衡数据下准确率96%,F1最高达94%,比微调更稳定
- 适合需合规管理、数据频繁更新的机密文档分类场景
未经授权披露机密文档风险高,需低泄漏、强鲁棒的分类方法。在真实工作环境中文档进出频繁,为持续更新知识,本文提出基于检索增强分类(RAC)的机密文档分类方法。在维基解密美国外交电报语料库上,于真实序列长度约束下对比RAC与监督微调(FT)。平衡数据下,RAC性能媲美FT;在不平衡数据下,RAC更稳定,原始(非平衡)与扩充(平衡)数据集上准确率均约96%,适当提示下F1最高达94%;而FT在平衡数据上得90% F1,但在原始非平衡数据上降至88% F1。当难以进行稳健数据增强时,RAC可在不暴露敏感内容、保持控制权的前提下实现强分类性能,且对类别分布、数据、上下文长度或治理要求变化具有鲁棒性。因决策基于外部向量存储的相似性匹配,RAC减少标签偏差影响,降低参数级泄漏风险,并可通过重新索引即时引入新数据——这对通常需重训练的FT而言困难。本文贡献包括:一、构建RAC分类流程与评估范式;二、在保密文档评分中隔离类别不平衡与上下文长度影响的对照研究;三、提供面向受控部署的RAC设计指南。
原文摘要 · Abstract (English)
Unauthorized disclosure of confidential documents demands robust, low-leakage classification. In real work environments, there is a lot of inflow and outflow of documents. To continuously update knowledge, we propose a methodology for classifying confidential documents using Retrieval Augmented Classification (RAC). To confirm this effectiveness, we compare RAC and supervised fine tuning (FT) on the WikiLeaks US Diplomacy corpus under realistic sequence-length constraints. On balanced data, RAC matches FT. On unbalanced data, RAC is more stable while delivering comparable performance--about 96% Accuracy on both the original (unbalanced) and augmented (balanced) sets, and up to 94% F1 with proper prompting--whereas FT attains 90% F1 trained on the augmented, balanced set but drops to 88% F1 trained on the original, unbalanced set. When robust augmentation is infeasible, RAC provides a practical, security-preserving path to strong classification by keeping sensitive content out of model weights and under your control, and it remains robust as real-world conditions change in class balance, data, context length, or governance requirements. Because RAC grounds decisions in an external vector store with similarity matching, it is less sensitive to label skew, reduces parameter-level leakage, and can incorporate new data immediately via reindexing--a difficult step for FT, which typically requires retraining. The contributions of this paper are threefold: first, a RAC-based classification pipeline and evaluation recipe; second, a controlled study that isolates class imbalance and context-length effects for FT versus RAC in confidential-document grading; and third, actionable guidance on RAC design patterns for governed deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。