124万参数小模型+8比特量化,实现离线法律条文精准检索
Quecto-V1: Empirical Analysis of 8-bit Quantized Small Language Models for On-Device Legal Retrieval
- 基于定制GPT-2架构,专训印度法律文本提升领域词密度
- 8比特量化后模型小于150MB,仅3.5%准确率损失下压缩74%
- 适合资源受限场景的法律从业者,保障数据隐私
大型语言模型虽推动自然语言处理发展,但其高算力需求导致资源鸿沟。现有法律智能系统多依赖70亿以上参数的云服务,难为资源有限者使用且存在数据主权风险。本文提出面向印度法律领域的小型语言模型Quecto-V1,基于124万参数的自定义GPT-2架构,从零训练于印度刑法典(IPC)、刑事诉讼法(CrPC)及宪法等法律文本。相比通用模型,该方法聚焦法律领域词汇密度。通过后训练8比特量化(GGUF格式),模型体积压缩至150MB以下。实证分析表明,其在法律条文定义与刑罚条款检索任务中表现优异,优于通用小模型,且可在消费级CPU上完全离线运行。消融实验显示,8比特量化使模型尺寸减少74%,准确率下降不足3.5%。结果表明,在法律等高风险专业领域,专用训练结合激进量化可成为隐私保护的可行替代方案。
原文摘要 · Abstract (English)
The rapid proliferation of Large Language Models (LLMs) has revolutionized Natural Language Processing (NLP) but has simultaneously created a "resource divide." State-of-the-art legal intelligence systems typically rely on massive parameter counts (7B+) and cloud-based inference, rendering them inaccessible to practitioners in resource-constrained environments and posing significant data sovereignty risks. This paper introduces Quecto-V1, a domain-specific Small Language Model (SLM) engineered to democratize access to Indian legal intelligence. Built upon a custom configuration of the GPT-2 architecture (124 million parameters), Quecto-V1 was trained from scratch exclusively on a corpus of Indian statutes, including the Indian Penal Code (IPC), the Code of Criminal Procedure (CrPC), and the Constitution of India. Unlike generalist models, which prioritize broad world knowledge, our approach maximizes "lexical density" within the legal domain. Furthermore, we address the deployment bottleneck by applying post-training 8-bit quantization (GGUF format), compressing the model to a memory footprint of under 150 MB. Our empirical analysis demonstrates that Quecto-V1 achieves high fidelity in retrieving statutory definitions and penal provisions, outperforming general-purpose SLMs in domain-specific exact match tasks while running entirely offline on consumer-grade CPUs. We further present an ablation study showing that 8-bit quantization yields a 74% reduction in model size with less than 3.5% degradation in retrieval accuracy compared to full-precision baselines. These findings suggest that for specialized, high-stakes domains like law, domain-specific training coupled with aggressive quantization offers a viable, privacy-preserving alternative to monolithic cloud models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。