Posts

A short pointer to the LMSYS article on work I contributed to: serving MOSS-TTS-Local Transformer v1.5 on SGLang-Omni with native streaming at 48 kHz stereo.

How we serve Ming-Omni in SGLang: the unified multimodal architecture, and the optimizations — an encoder kernel fix, tensor parallelism, CFM CUDA-graph capture, and streaming TTS — behind fast omni serving.

A short pointer to the LMSYS article on work I contributed to: serving Higgs Audio v3 TTS with SGLang-Omni for real-time, controllable voice agents.

Using Ming-flash-omni-2.0 as a case study to compare vllm-omni and sglang-omni design tradeoffs across thinker, talker, streaming, and audio output.

Why real multimodal model support is a six-layer serving-stack contract, from API extraction to decoder reentry.

How I designed, implemented, and hardened a cost-efficient RAG chatbot for my personal site with citations, streaming, and build-time indexing.

How gathering your information in one place transforms AI from a generic assistant into your personal superpower.

Why this feels like the shift from horsepower to a real vehicle—and why I am excited about the future.

Why vLLM requests can take 2-3 hours under heavy load - analyzing KV cache block exhaustion, queue starvation.

Continuing the Mini SGLang deep dive - covering request batching, overlap scheduling, and tensor parallelism.

Deep dive into Mini SGLang architecture - covering system design, engine initialization, KV cache, and single request lifecycle.

Prefill vs decode, continuous batching, quantization, speculative decoding, and CUDA graphs — a map of the LLM inference optimization landscape and what each technique actually buys.

How GPU memory actually gets spent serving an LLM: the PyTorch caching allocator, fragmentation, static KV pools, and debugging OOMs that make no sense.

Self-attention from an inference engineer's perspective: where the FLOPs go, why the KV cache exists, and how GQA and FlashAttention change the memory math.