用软加权融合提升长文本推理，解决碎片化问题。
Soft-NBCE: Entropy-Weighted Chunk Fusion for Long-Context
- 用熵加权软融合替代硬选择，实现跨块连续推理
- 在多跳问答任务上性能显著提升，如MuSiQue F1达0.310
- 轻量设计适合长上下文场景，内存开销仅为原方法的1/n
自注意力的二次复杂度仍是大语言模型处理超长上下文的瓶颈。朴素贝叶斯认知引擎（NBCE）通过分块文档并每步路由到最低熵块来并行长文本推理，但这种硬选择策略导致跨块推理时语义碎片化，相邻词元间路由突变破坏上下文连贯性。本文提出Soft-NBCE，一种轻量级扩展，将离散块选择替换为基于预测熵的软加权融合。通过温度缩放的Softmax为所有块分配连续权重，实现块条件分布的对数空间聚合。为缓解分块引入的条件独立性假设，提出一致性蒸馏：基于LoRA的自蒸馏机制，通过KL散度约束分块输出逼近全上下文教师模型。在LongBench多跳基准上，结合一致性蒸馏的Soft-NBCE持续优于基线（MuSiQue F1：0.310 vs. 0.275；HotpotQA F1：0.479 vs. 0.427），同时保持检索准确率（NIAH-32K：0.909），峰值内存降至O(L²/n)。
原文摘要 · Abstract (English)
The quadratic complexity of self-attention remains a bottleneck for Large Language Models (LLMs) processing ultra-long contexts. The Naive Bayes Cognitive Engine (NBCE) parallelizes long-context inference by chunking documents and routing to the lowest-entropy chunk at each decoding step. This hard-selection strategy causes semantic fragmentation during cross-chunk reasoning, as abrupt routing changes between adjacent tokens disrupt the model's contextual grounding. We present Soft-NBCE, a lightweight extension that replaces discrete chunk selection with soft entropy-weighted chunk fusion. A temperature-scaled Softmax over predictive entropies assigns continuous weights to all chunks, enabling log-space aggregation across chunk-conditioned distributions. To partially compensate for the conditional independence assumption introduced by chunking, we propose Consistency Distillation, a LoRA-based self-distillation that constrains the chunked logit distribution toward a full-context teacher via KL-divergence. On LongBench multi-hop benchmarks, Soft-NBCE with Consistency Distillation improves consistently over NBCE-style baselines (MuSiQue F1: 0.310 vs.\ 0.275 for Vanilla NBCE; HotpotQA F1: 0.479 vs.\ 0.427) while maintaining retrieval accuracy (NIAH-32K: 0.909) at O(L^2/n) peak memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。