Why the Best Inference Configuration Can Fail in Production
An interactive guide to the workload, memory, routing, and capacity assumptions behind an inference benchmark winner.
I work on LLM and multimodal inference.
My work includes model integration and serving optimizations in SGLang and vLLM. Here I write about implementation details, performance bottlenecks, and the tradeoffs behind benchmark results.
Featured
MOSS-TTS Local Transformer v1.5 on SGLang-Omni
A short pointer to the LMSYS article on work I contributed to: serving MOSS-TTS-Local Transformer v1.5 on SGLang-Omni with native streaming at 48 kHz stereo.
Ming-Omni on SGLang: Architecture and the Optimizations Behind Fast Omni Serving
How we serve Ming-Omni in SGLang: the unified multimodal architecture, and the optimizations — an encoder kernel fix, tensor parallelism, CFM CUDA-graph capture, and streaming TTS — behind fast omni serving.