My main interest and focus is on machine learning systems and large language model inference — serving models, managing KV cache, squeezing latency out of GPUs. I also contribute to open-source inference engines like SGLang and vLLM. This site is where I write down what I learn, lessons and the mistakes along the journey.
Check my blogs and projects below!
How we serve Ming-Omni in SGLang: the unified multimodal architecture, and the optimizations — an encoder kernel fix, tensor parallelism, CFM CUDA-graph capture, and streaming TTS — behind fast omni serving.
A short pointer to the LMSYS article on work I contributed to: serving Higgs Audio v3 TTS with SGLang-Omni for real-time, controllable voice agents.
Using Ming-flash-omni-2.0 as a case study to compare vllm-omni and sglang-omni design tradeoffs across thinker, talker, streaming, and audio output.