All the articles I've posted.
A short pointer to the SGLang article on work I contributed to: a unified SGLang-Omni runtime for native full-duplex, turn-based, and cascaded interactive systems, evaluated under one real-time interface.
A short pointer to the SGLang article on work I contributed to: the Stage–Process–GPU topology that makes stage grouping, replication, and GPU placement a deployment-time configuration in SGLang-Omni.
An interactive guide to the workload, memory, routing, and capacity assumptions behind an inference benchmark winner.
A short pointer to the LMSYS article on work I contributed to: serving MOSS-TTS-Local Transformer v1.5 on SGLang-Omni with native streaming at 48 kHz stereo.
How we serve Ming-Omni in SGLang: the unified multimodal architecture, and the optimizations — an encoder kernel fix, tensor parallelism, CFM CUDA-graph capture, and streaming TTS — behind fast omni serving.
A short pointer to the LMSYS article on work I contributed to: serving Higgs Audio v3 TTS with SGLang-Omni for real-time, controllable voice agents.
Using Ming-flash-omni-2.0 as a case study to compare vllm-omni and sglang-omni design tradeoffs across thinker, talker, streaming, and audio output.
Why real multimodal model support is a six-layer serving-stack contract, from API extraction to decoder reentry.
How I designed, implemented, and hardened a cost-efficient RAG chatbot for my personal site with citations, streaming, and build-time indexing.
How gathering your information in one place transforms AI from a generic assistant into your personal superpower.
Why this feels like the shift from horsepower to a real vehicle—and why I am excited about the future.
Why vLLM requests can take 2-3 hours under heavy load - analyzing KV cache block exhaustion, queue starvation.
Continuing the Mini SGLang deep dive - covering request batching, overlap scheduling, and tensor parallelism.
Deep dive into Mini SGLang architecture - covering system design, engine initialization, KV cache, and single request lifecycle.
Prefill vs decode, continuous batching, quantization, speculative decoding, and CUDA graphs — a map of the LLM inference optimization landscape and what each technique actually buys.
How GPU memory actually gets spent serving an LLM: the PyTorch caching allocator, fragmentation, static KV pools, and debugging OOMs that make no sense.
Self-attention from an inference engineer's perspective: where the FLOPs go, why the KV cache exists, and how GQA and FlashAttention change the memory math.