用证据对齐的局部专家组合修复被损坏的文本序列。
Evidence-Aligned Local Composition of Discrete Experts for Sequence Restoration
- 根据噪声模型推断每个位置的专家软权重，无需训练路由器。
- 在自然混合科学文档上达到0.85的区域识别准确率，离散词段上达0.98。
- 适合处理多领域混合文本修复，尤其适用于专家差异明显时。
将文档视为由不同领域文本组成的离散标记序列，例如README文件在说明文、代码和配置间切换。当文档受损且仅可用冻结的领域专家时，恢复需在测试阶段决定缺失内容及各位置应信任的专家，无区域标签或训练过的路由机制。本文提出证据对齐的局部组合方法：基于给定噪声模型下受损观测的边际证据，从专家自身的去噪损失中估计证据，并在位置间平滑权重。由于权重为软分配，当真实组合为混合时可恢复混合结果，当单一专家足够时则集中于该专家。在分类模拟器、字节级专家及基于1.3B离散流匹配模型微调的专家上，推断权重在自然混合科学文档上达到0.85的字段准确率，在词汇不交叠的构造混合样本上达0.98。修复效果优于单一全局权重，当专家趋同则退化为全局权重，能追踪专家分离程度。
原文摘要 · Abstract (English)
A document modeled as a discrete sequence of tokens can be thought of as being generated from a composition of texts from different domains; a README file, for example, moves between prose, code, and configuration. When such a document is corrupted and only frozen domain experts are available, restoring it requires deciding both what is missing and which expert to trust at each position, at test time and without region labels or a trained router. We introduce evidence-aligned local composition, which infers a soft, position-wise weighting over the experts from the marginal evidence of the corrupted observation under a given corruption model, estimating the evidence from the experts' own denoising losses and smoothing the weights across positions. Because the weighting is soft, it recovers a mixture when the true composition is mixed and concentrates on one expert when that suffices. Across a categorical simulator, byte-level experts, and experts fine-tuned from a $1.3$B discrete flow-matching model, the inferred weights track the true regions at $0.85$ field accuracy on naturally mixed scientific documents, and at $0.98$ on constructed mixtures whose regions are lexically disjoint. Restoration improves over a single global weight when the experts are genuinely distinct and reduces to it when they converge, tracking a measure of expert separation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。