GitHub / SGLang: релиз v0.5.18

Highlights 710 PRs from 212 contributors. New models in this release (see the cookbook for all supported models): Model Type PRs Cookbook Muse Glimmer Autoregressive (Multimodal) #34262 link Intern-S2-Mobius Autoregressive #33691 link SANA-Video Diffusion #32921 link LingBot-Video-MoE Diffusion #32341 link LTX-2.5 Diffusion #34471 link Cosmos3 Edge & Distilled Diffusion #31590 link LongCat-Image Diffusion #23274 Plus cookbook recipes for the Qwen3.8 family , Ling-3.0 , Nemotron 3.5 Lightning , Dots3-Note , and DeepSeek-V4-Pro-0813 ( #34809 ). Overlapped checkpoint staging at startup : Checkpoint pages now stage from storage while CUDA graphs capture. Qwen3-32B on H100 starts 8.6-11.7% faster than serial with prefetch, and 2.38x faster (35.6s vs 84.8s) than the plain default. Opt in with --startup-weight-load-mode overlap ( #32017 ). TP LMHead with All-to-All : The TP LMHead's allgather + scatter becomes a single all-to-all for pure-DP dp-attention. On DeepSeek-V4-Pro B200 decode, LMHead time drops 320us to 169us and TPOT improves 36.97ms to 35.67ms ( #32313 ). FlashInfer MNNVL for pure allreduce : Non-fused allreduce sites now reuse the FlashInfer MNNVL workspace instead of falling back to NCCL. DeepSeek-V4-Flash TP4 decode on Blackwell gains up to +6.9% at small batches . Auto-enabled for DeepSeek-V3/V3.2/V4; elsewhere --enable-flashinfer-pure-allreduce ( #30700 ). NVFP4 checkpoints run on AMD : --quantization quark_mxfp4 dequantizes ModelOpt and Quark NVFP4 weights and requantizes to MXFP4 at load, never holding a full-precision copy. 97.5-100.2% GSM8K recovery vs the NVFP4 reference across MiniMax-M2.7, GLM-5.1, Kimi-K2.6, Qwen3.5-397B, and DeepSeek-R1 ( #29328 ). Kimi K3 tuned for MI355X : a grouped-head MLA verify kernel replaces the MHA-shaped split-KV path that re-read the shared latent once per head, for 1.37-1.77x throughput and 1.45-2.42x ITL at concurrency 2-32; the AITER MLA prefill kernel now accepts K3's 12-head shape (TTFT up to -14.9%); a gfx950-tuned decode stage-1 geometry adds 46-73% ITL at 68k input, opt-in via SGLANG_MLA_DECODE_TUNE=1 . GSM8K holds at 0.951-0.957 ( #33981 , #34261 , #34837 , #34580 ). One compiled-kernel cache directory : Triton, FlashInfer, Inductor, DeepGEMM, and CUDA driver caches all move under SGLANG_CACHE_DIR . The first launch after upgrading recompiles once; see Breaking Changes ( #32434 ). Dependencies : torch 2.13.0 with triton 3.7.1 ( #28836 ), flashinfer 0.6.17 ( #33997 ), CuTeDSL 4.6.2, fixing an FA4 startup regression on Blackwell ( #34372 ), DeepEP now installed from released sgl-deep-ep wheels ( #33932 ), and sgl-kernel 0.4.6.post1 ( #33842 ). Full release notes by category below; breaking changes and known issues are at the end. Rust Server [mm] rust-server: native multimodal processing for Qwen VL (integrate sglang-mm, e2e): #32365 refactor error responses into shared utils::response helpers: #33894 move the PD bootstrap registry under api_server::disaggregation: #33895 Build Rust extensions on demand in source checkouts: #34994 Speculative Decoding [spec decoding] support inkling dspark: #31847 [Spec] Support logprobs with DSpark speculative decoding: #34696 [Spec] Support output logprobs with DSpark: #34478 [Spec] Support logprobs with DFlash: #33459 [Spec] Wire DFLASH aux-hidden capture into the Qwen3.5 text-only wrapper: #34771 [Spec] Support mamba-radix-cache-strategy extra_buffer_lazy with DFLASH: #34763 [Spec] Support MegaMoE for DSpark under dp attention: #34844 [unified memory] Support DSPARK speculative decoding + fix two NaN root causes (page hand-out zeroing, CuTe int32 slot-stride wrap): #33974 [GDN] Honor configured linear-attn verify backend in the kernel dispatcher: #34592 [Spec] Relay ngram accept tokens through the FutureMap: #35198 [Spec] Reduce host-side overhead in ngram draft prep: #35207 [Spec] Point multi-layer eagle's last shared-read runner at the draft runner: #35057 Fix DFlash sliding attention causality defaults: #34524 [Spec] Budget the DFLASH draft KV pool from its own attention geometry: #34234 fix(dflash): account for DCP in draft KV pool sizing: #33912 [DSV4] Fix silent KV corruption when speculative draft tokens > 4: #34189 [Fix] Speculative decoding crashes with DP-Attention: #33892 [bugfix] Stop/EOS inside a spec accept run beats the max_new_tokens finish: #33758 [DSpark] Fix EP1 decode performance regression: #34759 [Fix: RL] Snapshot async state-capture outputs before overlap: #34319 Piecewise & Breakable CUDA Graph [BCG][5/N] MLA Fully Support: #33661 [BCG][6/N] Allow prefill breakable CUDA graph for the Kimi archs: #34245 fix: always capture default prefill CUDA graph: #33352 Fix padded positions in breakable CUDA Graph attention: #33253 fix: avoid piecewise prefill graph for trtllm_mla: #32785 Reenable breakable CUDA graph for NemotronH: #34538 Fix prefill CP graph overflow with larger bucket search: #33906 Fix stale track rows corrupting conv checkpoints under the prefill graph: #34184 Fix sconv track refresh on graph capture: #35042 Increase post-capture decode memory reserve: #34996 Attention Backends feat(attention): add architecture-owned SM12x FA4 kernels: #32991 fix: support FA4 backend for GLM4.7-flash: #33436 feat: Add flashinfer mHC fusion for DSV4: #33616 [DSV4] Turn on mhc post pre fusion by default: #35214 [SM12x] Default the fused MHC post+pre path on: #34019 [trtllm_mha] perf: Stop allocating per-layer scratch inside the decode CUDA graph: #33063 Select DeepGEMM standard layouts by memory budget: #33474 [NVIDIA] Enable CuTe DSL BF16 GEMM on SM107: #33617 add flashinfer cute-dsl backend for mxfp8 gemm: #34042 [BugFix] Fix race in c128 prefill plan kernel on ragged extend: #32467 [JIT Kernel][DSv4] Optimize epilogue of c128: #26671 fix(dsa): use FlashInfer fused top-k for packed PAGED rows: #33006 [DSA] Fix top-k v2 dropping non-primary ranks' output on CUDA 13.1+ (root cause for #33835 ): #34167 [DSA] Skip indexer KV cache for skip-topk layers: #30531 [Perf] Skip trivial DSV4 nonpaged indexer logits: #33857 [Fix] Restore layer-level DSV4 RoPE policy: #34788 Fix _pa_swa_prefill_lens off-by-one in FlashAttentionBackend: #33379 Support unified SWA page mapping in attention metadata: #35000 MoE & Expert Parallelism [MoE] Fuse swiglu moe up gemm epilogue: #32944 Enable MoE deferred finalize by default and drop its expert_weights dtype workaround: #33618 [kernel] add triton moe TMA up support: #33559 [MoE] Single-launch moe_align for tiny batches with many experts: #32395 [MiniMax-M3] Overlap shared and routed experts: #34542 [MoE] Route every trtllm-gen MoE call site through one PDL guard: #34789 fix(qwen3): support DeepEP-class backends and early EPLB state: #34810 Add explicit EPLB balancedness reporting modes: #34998 Fix the router GEMM inaccuracy when using _front_w in Kimi-K3: #33764 [Kimi-K3] Use explicit SiTU activation for MegaMoE: #34883 Fix Qwen3-MoE producing garbage with the mori a2a backend: #34006 [Fix][Qwen]: fused shared-expert detection PP-safe protection: #34447 [FlashInfer v0.6.16] Support FlashInfer CuTe DSL NVFP4 MoE quantization: #28354 [Fix] Correct W4AFP8 DeepEP scaling and mode-specific dtypes: #33669 [Fix] Support 128-aligned hidden sizes in the W4AFP8 DeepEP low-latency requant kernel: #35593 perf: add H200 Triton MoE configs for E256 N512: #34150 Quantization Support ModelOpt MXFP8 checkpoints: #32538 [ModelOpt FP4] Support online MoE weight quantization: #33115 [Feature] Support NVFP4 token embedding in ModelOpt mixed-precision checkpoints: #34222 [MXFP8] Use FlashInfer CUTLASS for dense GEMM on SM120, delete Triton path: #33208 [Quantization] Route per-tensor FP8 checkpoints to FlashInfer on SM90: #33148 Optimize FP32 LM head for bf16/fp16: #32370 Pin online NVFP4 4over6 quantization settings: #33621 [Fix] Support Kimi-K3 ModelOpt mixed NVFP4/FP8 checkpoint: #35077 [Quantization] Fix GPTQ scheme attachment broken by LinearBase.scheme default: #34962 [quantization] Add tuned Triton tile configs for channelwise FP8 GEMM…: #34331 [Fix] Correct dense FP8 Marlin bias ordering: #35020 Remove the torchao integration (--torchao-config): #34304 Parallelism & Disaggregation [Feature] Optimize TP LMHead with All-to-All: #32313 ⭐ [NVIDIA] Add flashinfer MNNVL backend for allreduce only: #30700 ⭐ [CP]: Support CP V2 Strategy for dsv4: #33532 fix: preserve GQA head mapping in Triton DCP prefill: #34161 Fix DCP KV head mapping for GQA models: #32858 [DCP] Reuse partial output in natural-log LSE merge: #34213 [DCP] Drop two per-layer launches from the MLA target-verify path: #34240 [DCP] Fuse the a2a pack/unpack copies in the MLA LSE reduce: #34614 [DCP] Share one pack kernel between both a2a backends: #34651 [DCP] Drop the prefill index-selection syncs by taking each rank's rows by stride: #35084 [Fix] DCP: advertise the logical KV-event block size: #35298 Fix DeepSeek-V4/DeepSeek-V4-Pro DP-attention gather semantics: #31700 [perf] Collapse the DP attention scheduler sync to a single D2H copy: #34338 Multiple flexibility fixes for DP attention: #33537 Fix world-size-one aliasing in MLP batch sync: #34997 [PD] Add the missing Prefill bootstrap timeout for NIXL: #34692 [PD] Support pipeline-parallel prefill with Mooncake staging buffer: #33807 [PD] Support --enable-unified-memory with PD disaggregation (kimi-linear MLA hybrid-Mamba): #33362 [PD] Skip speculative verify scratch on prefill servers (saves num_draft_tokens x mamba pool per rank): #34191 [PD] Avoid unused PREBUILT prompt tensor transfer: #35070 feat(grpc): expose KV event discovery metadata: #35714 Scheduler & Runtime [Model Loading] Overlap checkpoint staging with CUDA graph capture during startup: #32017 ⭐ [core] Consolidate compiled-kernel caches under SGLANG_CACHE_DIR: #32434 ⭐ [Engine] Freeze GC after server warmup: #34999 feat: add cache salt support to KV cache events: #30827 perf(kv-events): coalesce cache events: #31479 [Scheduler] Honor explicit min-free-slots thresholds: #33403 [Scheduler] Gate SWA eviction on accumulated tokens: #33404 [Scheduler] Fix to restrict the SWA chunk-cap escape hatch to true head-of-line livelock: #32700 [srt] Batch scheduler cache frees: #33475 O(1) slot allocation in ReqToTokenPool.alloc(): #32208 fix(scheduler): track max prefill batch size over recent real admissions: #34284 fix tpot by adjusting the sliding max-prefill-size window size: #34856 fix: route scheduler aborts to multi-tokenizer workers: #33940 [tracing] sglang tracing v2: support exporting tracing data asynchronously: #30023 Profiling Enhancements [1/3]: cuda graph profile traces: #24370 [metrics] Fix prefill FLOPs estimate to count prefix and per-request causal pairs: #34316 Fix default dtype restoration after model loader errors: #34440 fix: honor explicit model loader classes: #34880 feat(cli): add extensible serve backend plugins: #34753 Clean up environ.py: remove dead env vars, unify deprecation handling, move examples to a unit test: #35060 fix: make automatic NUMA binding configurable: #30394 Raise PD zmq per-context socket cap via SGLANG_DISAGGREGATION_ZMQ_MAX_SOCKETS: #34450 Deterministic Inference Fix deterministic inference for Inkling: #33417 Fix deterministic inference all-reduce for tp>1: #34159 Deterministic gumbel sampling: clamp u=1 so masked tokens can't be sampled: #33423 feat: support deterministic FA4 for GLM-4.7-Flash: #33945 Fix NaN logits from deterministic Triton extend on the unified memory pool: #33517 Add bit-exact class for MTP: #35143 Add bit-exact unified radix cache KL test for hybrid SWA + mamba: #34607 Add bit-exact hicache logprob-consistency test: #34356 Add bit-exact guard for extra_buffer_lazy: #35030 Skip inkling sheared bias under batch invariance: #35161 HiCache & Radix Cache [HiCache] Support packed and sidecar draft caches for MTP/EAGLE/DSpark: #30393 [Hicache][2/2]Support Mamba branching in Unified Radix Cache with HiCache: #33639 [HiCache] Support DCP with DSpark: #35221 Enable unified cache out-of-window slot freeing by default: #34653 Skip oow slot freeing under eagle: #34823 [HiCache] write_back: reclaim duplicated host copy first under host pressure: #33777 [HiCache] Optimize LogicalHostPool free-list release: #33998 refactor(hicache): flatten L2 transfer execution: #34793 [PD] Preserve decode KV across retraction in HiCache: #34801 fix(hicache/umbp): support DeepSeek-V4 hybrid HostPoolGroup (multi-po…: #30762 [Unified Radix Cache] Complete the tree-core interface boundary: #33580 [Fix] Snapshot req.prefix_indices when the prefix cache is disabled: #34644 Fix swa eviction frontier for bigram keys: #34870 Retain SWA down to the last state checkpoint: #34729 Fix mamba checkpoint depth under dcp: #34808 HiSparse: shared-index (IndexShare) plan-then-IO swap-in prefetch: #34329 Observability enhancement for HiCache: #32388 Multimodal [VLM] split multimodal scheduling from mm_utils: #32415 [VLM] add content-addressed preprocessing cache infrastructure: #34398 Add CUDA VMM multimodal feature transport: #33899 feat(vlm): auto-select cuda vmm on multi-node mnnvl: #33936 Fix CUDA 13.0 VMM handle type compatibility: #34431 Gate multimodal feature transport by model capability: #33653 Reserve multimodal runtime allocations and keep padded inputs aligned: #34141 [VLM] Avoid synchronizing multimodal placeholder counts: #34995 fix(vlm): stream-order cuda-ipc feature pool lifecycle and streamline multimodal transport module: #33949 [EPD] feat: pipeline owner-only multimodal preprocessing: #34206 [FEAT] Decouple multimodal global cache from Mooncake: #30392 [Kimi K3] optimize: preprocess cpu-transport images on the vision owner: #33921 fix(vlm): preserve Kimi-K3 GPU JPEG accuracy: #34163 vlm: cache kimi-k3 per-image processor artifacts: #34404 [Fix] Carry the backend on Kimi-K3 deferred preprocessing configs: #34766 Fix Whisper transcription for audio over 30 seconds: #33604 fix: preserve priority for batched embedding requests: #32977 Model Support & Optimizations fix(muse-glimmer): parse required/named tool calls natively: #34781 Responses support: #32689 [Frontend] Apply request header overrides to chat completions: #35001 Stop losing Kimi-K3 tool calls to reasoning, constraint conflicts, and truncation: #34881 [Fix] Work around xgrammar 0.2.1 negative integer minimum in Kimi-K3 structural tags: #34778 [Fix] Make DeepSeek-V4 reasoning and tool-call streaming parsing chunk-invariant: #34458 [Fix] Require JSON booleans for response_format json_schema.strict: #34777 Support thinking budget for Inkling: #33146 [inkling] Let Anthropic thinking=disabled map to reasoning effort "none": #33913 [inkling] Render tool-result media instead of coercing content to str: #33898 [DSV4] Add official DSV4 reasoning effort support: #33140 [feat] Add language_model_only parameter support for Qwen35: #22867 Support model-defined prefill input embedding width: #35002 Fix rope config compatibility and VL/transformers-fallback weight loading: #31575 [LoRA] Laguna: per-layer LoRA hid 💡 Где захостить и как запустить: рассчитать точные требования к объёму VRAM под разные типы квантования этой модели (4-bit Q4, FP8, FP16), оценить скорость генерации токенов в секунду и подобрать совместимый сервер с GPU можно на ServerSales в интерактивном подборщике AI-моделей или каталоге GPU-серверов.