arXiv:2509.22211cs.CLcs.AI2025-09

用本地大模型高效构建可解释的文本分类体系,支持大规模数据探索。

LogiPart: Local Large Language Models for Data Exploration at Scale with Logical Partitioning

  • 通过局部大模型生成自然语言分类规则,降低计算成本。
  • 在14万文档上实现96%的分类准确率,优于传统主题模型。
  • 可在普通电脑上运行,适合需要可解释性的研究者使用。

在大规模文本语料中发现深层、可调控的分类体系,目前受限于主题模型的效率与集成大模型框架的高昂计算开销之间的权衡。我们提出 extbf{LogiPart},一种可扩展的、以假设为先的框架,用于构建可解释的层次化分类结构,将层级增长与昂贵的全语料大模型推理解耦。LogiPart 在小型嵌入感知样本上使用本地部署的大模型生成简洁的自然语言分类谓词,再通过零样本自然语言推理(NLI)与快速图传播算法,在整个语料库中高效评估这些谓词,实现相对于语料规模恒定的 $O(1)$ 每节点生成令牌复杂度。我们在四个不同文本语料(总计约14万篇文档)上评估了LogiPart。通过结构流形进行校准,我们识别出140亿参数量级是实现稳定语义锚定的经验阈值。在高熵复杂语料(如Wikipedia、美国法案)中,传统主题指标显示存在“对齐差距”,但逆向逻辑验证确认了所生成逻辑的稳定性,单个分类分割的平均节点路由准确率达96%。独立大模型作为评判者进行定性审计,确认发现了政策意图等语义轴线,而传统标签无法捕捉。LogiPart 实现了在消费级硬件上的前沿探索分析,使在真实算力和治理约束下进行假设驱动的分类发现成为可能。

原文摘要 · Abstract (English)

The discovery of deep, steerable taxonomies in large text corpora is currently restricted by a trade-off between the surface-level efficiency of topic models and the prohibitive, non-scalable assignment costs of LLM-integrated frameworks. We introduce \textbf{LogiPart}, a scalable, hypothesis-first framework for building interpretable hierarchical partitions that decouples hierarchy growth from expensive full-corpus LLM conditioning. LogiPart utilizes locally hosted LLMs on compact, embedding-aware samples to generate concise natural-language taxonomic predicates. These predicates are then evaluated efficiently across the entire corpus using zero-shot Natural Language Inference (NLI) combined with fast graph-based label propagation, achieving constant $O(1)$ generative token complexity per node relative to corpus size. We evaluate LogiPart across four diverse text corpora (totaling $\approx$140,000 documents). Using structured manifolds for \textbf{calibration}, we identify an empirical reasoning threshold at the 14B-parameter scale required for stable semantic grounding. On complex, high-entropy corpora (Wikipedia, US Bills), where traditional thematic metrics reveal an ``alignment gap,'' inverse logic validation confirms the stability of the induced logic, with individual taxonomic bisections maintaining an average per-node routing accuracy of up to 96\%. A qualitative audit by an independent LLM-as-a-judge confirms the discovery of meaningful functional axes, such as policy intent, that thematic ground-truth labels fail to capture. LogiPart enables frontier-level exploratory analysis on consumer-grade hardware, making hypothesis-driven taxonomic discovery feasible under realistic computational and governance constraints.

大模型文本分类可解释性高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。