Systems engineer — enterprise infrastructure by trade, GPU inference systems by obsession. Fifteen years building across the stack: Linux, virtualization, storage, HA networking, and embedded — with a deep self-directed focus on high-performance LLM serving.
I like problems where the answer is "that shouldn't work on this hardware."
I build heterogeneous-GPU inference — running large models across cards of different generations, VRAM sizes, and interconnects that stock engines either can't combine at all or throttle to the weakest card. Two forks, same core idea, on both major serving engines:
shvllm — a vLLM fork for mismatched GPUs. Uneven tensor parallelism: every sharded dimension is partitioned by a per-rank weight vector, so cards of different sizes (an RTX 5090 next to two RTX 3080s, PCIe-only, no NVLink) can run a single TP group. Makes TP possible on models whose head counts don't divide evenly, self-calibrates the KV cache across measured VRAM, and ships hand-tuned MMQ/MMVQ decode kernels with bit-identical outputs.
- ~87 tok/s decode and a 1,037,653-token KV pool on 72 GB of mixed consumer cards
- Verified with 4 concurrent ~250k-context sessions, zero preemptions
- Reproducible via public Docker container + full benchmarks
- Companion vllm-gguf-plugin: GGUF support with MTP speculative decoding
- Includes HTCCL, a vendor-neutral collectives layer (host-staged, CUDA-graph-capturable, per-link bandwidth-weighted) reaching NCCL parity on P2P-less hardware — so a future mixed AMD + NVIDIA box can run a single TP group, where NCCL doesn't go. On pure-NVIDIA setups you just use NCCL.
- Qwen3.6-27b tested only
htsglang — the same for SGLang. I'd built this for vLLM and it was missing on the stronger engine, so I brought it over — and went further. Uneven TP again, plus true multi-rank-per-GPU co-location (NCCL 2.30+), native NEXTN/MTP speculative decoding, and a 3-tier KV cache that spills GPU → host RAM → disk (RadixAttention prefix dedup keeps long shared prefixes resident and re-hydrates them instead of recomputing).
- Qwen3.6-27b tested only, more incoming
Both are hobbyist forks moving fast, with honest documented limitations (single-node TP only, whole-unit head/group splits). Upstream contributions in community review — features complete, gathering real-world test results before merge.
- Active/passive HA cluster-in-a-box over iSER (RDMA storage) — AP-HA-CIAB-ISER
- Real-time video upscaling + frame interpolation (private) — VapourSynth integrated into FFmpeg for TensorRT-based processing, wired into Jellyfin
- "Gustav," a voice assistant for my kids (absolute private) — a ghost from a children's book they can actually talk to. Extracted one character's voice from a TV series, isolated the speaker segments with ML, cleaned the audio, fine-tuned an open-source TTS voice on it, and deployed the whole thing to ESPHome with a display animation. Full pipeline, built purely because my kids wanted to talk to gustav.
- LMCache extension LMCache Fork to eliminate VRAM usage
LLM inference (vLLM · SGLang · CUDA kernels) · Virtualization (Proxmox · vCenter/ESXi · KVM · LXC) · Containers (Docker · Portainer) · Storage & HA (iSER · SAN · ZFS/TrueNAS) · Networking (VLANs · multi-site) · Languages (Python · C · Shell)
I use AI coding agents as an amplifier on kernel-level work — output quality is the part that matters, and that's where the systems depth lives.