智能调度器动态优化数据处理延迟与内存占用
Adaptive Execution Scheduler for DataDios SmartDiff
- 根据资源预算自动调节批处理大小和线程数
- 降低95%延迟23%-28%，内存峰值下降16%-22%
- 适合需要低延迟、高稳定性的数据流水线场景
我们提出一种针对单一差异引擎（SmartDiff）的自适应调度器，支持内存线程与基于Dask的并行两种执行模式。调度器在固定CPU和内存预算下，持续调整批处理大小与工作线程数，以最小化p95延迟。轻量级预检分析器估算每行字节数与I/O速率；在线成本/内存模型用于剔除不安全操作；受保护的爬山策略结合背压与慢节点缓解机制，优先选择更低延迟。后端选择由保守的工作集估计决定：当安全时优先使用内存执行，否则启用Dask。在合成及公开表格基准测试中，相比调优的预热启发式方法，该调度器使p95延迟降低23%至28%（相比固定网格基线降低35%至40%），同时峰值内存降低16%至22%（相比固定网格降低25%至32%），无任何内存溢出（OOM），吞吐量相当。
原文摘要 · Abstract (English)
We present an adaptive scheduler for a single differencing engine (SmartDiff) with two execution modes: (i) in-memory threads and (ii) Dask based parallelism. The scheduler continuously tunes batch size and worker/thread count within fixed CPU and memory budgets to minimize p95 latency. A lightweight preflight profiler estimates bytes/row and I/O rate; an online cost/memory model prunes unsafe actions; and a guarded hill-climb policy favors lower latency with backpressure and straggler mitigation. Backend selection is gated by a conservative working-set estimate so that in-memory execution is chosen when safe, otherwise Dask is used. Across synthetic and public tabular benchmarks, the scheduler reduces p95 latency by 23 to 28 percent versus a tuned warm-up heuristic (and by 35 to 40 percent versus fixed grid baselines), while lowering peak memory by 16 to 22 percent (25 to 32 percent vs. fixed) with zero OOMs and comparable throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。