The article tackles what single-loop LLM serving engines were never built for: a request to an omni, speech, or TTS model traverses heterogeneous stages — modality encoders, a thinker, a talker, and a streaming vocoder — that differ in bottleneck resource, resident state, and latency tolerance. How those stages are grouped into processes, replicated, and placed on devices is the execution strategy, and the article argues it affects serving performance as much as the efficiency of any individual stage. SGLang-Omni expresses it as a deployment-time Stage–Process–GPU (SPG) topology instead of per-model code.
The useful framing is three measurable characteristics that predict a good layout: the synchronization granularity of each dependency (one-shot handoff, bufferable stream, or per-step feedback), and each stage's bottleneck resource and resident state size. Feedback-coupled fusion, process grouping, replication, encoder–prefill–decode disaggregation, and stage offload then become instances of one representation that the same runtime can realize without model-specific changes.
A lot of my Qwen3-Omni serving work lives at exactly this boundary — for example, placing code2wav on the thinker's GPU so the talker gets its own device (sglang-omni#1235), and bounded wait/floor batching plus exact-shape CUDA graph replay for the Code2Wav vocoder (sglang-omni#1126, sglang-omni#1101).
Read the full article on SGLang.