The article tackles a serving problem that outlives any single generation request: continuous voice interaction, where the system has to keep listening while it speaks, decide when to respond or yield, and stop cleanly on interruption. SGLang-Omni adds a graph abstraction that composes models, tools, and interaction control, plus persistent sessions that coordinate lifecycles and events across a conversation. The same runtime serves a native full-duplex model (MiniCPM-o 4.5), a turn-based omni model (Qwen3-Omni with VAD turn detection), and a cascade of Qwen3-ASR, Qwen3.8-27B, and Qwen3-TTS — all on one H200 each.
The evaluation shows that no approach dominates. The turn-based model and the cascade stop within about a quarter of a second of an interruption, but treat speech not meant for them as a new turn; MiniCPM-o keeps talking through side conversations and background speech but takes about 1.6 s to stop. The cascade sustained 16 concurrent sessions per GPU at its admission limit, versus 8 for MiniCPM-o and 2 for the single-GPU Qwen3-Omni layout. Measuring behavior, serving capacity, and speech quality through one interface is what makes those trade-offs comparable.
I worked on the full-duplex evaluation and serving path: the reproducible full-duplex evaluation (sglang-omni#2331), concurrent session recording so latency is measured under real load rather than one session at a time (sglang-omni#2703), keeping recorder pacing within 80 ms at 24–64 sessions (sglang-omni#3288), and keeping interrupt acknowledgments flowing while a reply is being preempted (sglang-omni#3287).
Read the full article on SGLang.