A high-performance API server that provides OpenAI-compatible endpoints for MLX models. Developed using Python and powered by the FastAPI framework, it provides an efficient, scalable, and user-friendly solution for running MLX-based vision and language models locally with an OpenAI-compatible interface.

LLM inference engine from scratch — paged KV cache, continuous batching, chunked prefill, prefix caching, speculative decoding, CUDA graph, tensor parallelism, OpenAI-compatible serving

A High-Performance LLM Inference Engine with vLLM-Style Continuous Batching

An Efficient and Versatile Inference Engine for Distributed LLM Serving

A lightweight, educational LLM inference engine for studying continuous batching, paged KV cache, chunked prefill, and online serving.

Continuous batching for TTS — like vLLM, but for voice. Serve 10+ simultaneous text-to-speech requests on a single GPU.

An AI inference engine from scratch. Like nanoGPT, but for serving.

An educational LLM inference engine built from scratch: custom decoding loop, KV cache, continuous batching, paged memory with preemption, prefix caching, speculative decoding, OpenAI-compatible API, real benchmarks.

High-Performance LLM Inference Engine with PagedAttention & Continuous Batching in Rust

Local-first LLM stack on a single RTX 5090: QLoRA fine-tuning, exact speculative decoding, paged KV-cache, and continuous batching — served via FastAPI with a live React dashboard.

A production-grade, native Rust speculative inference engine for Apple Silicon with Metal GPU acceleration and paged attention.

🌱 A tiny, readable LLM serving engine with vLLM/SGLang-style features.

W8A8 (INT8 dp4a) inference server for NVIDIA Volta / CMP 100-210, over the fni8 FlashAttention-2 kernels

Real PyTorch inference server comparing FCFS, EagerContBatch, and ChunkedPrefill on RTX 2070. Key finding: ChunkedPrefill is not a throughput optimizer but a fairness mechanism — it reduces worst-case decode stalls from 58.6ms to 34.1ms while EagerContBatch maximizes mean TTFT and throughput.

A from scratch LLM inference engine build in PyTorch with custom GPT2 transformers, kv cache, paged kv cache, continuous batching and A100 benchmarks

Fork of OpenAI and Anthropic compatible server for Apple Silicon. Native MLX backend, 500+ tok/s. Run LLMs and vision-language models with continuous batching, MCP tool calling, and multimodal support.

A minimal LLM inference server with a real paged KV-cache, continuous batching and speculative decoding, by sarmalinux

Discrete-event simulator of LLM inference scheduling policies, comparing FCFS, continuous batching, priority, SLO-aware, and chunked prefill under memory pressure and latency SLOs.

OpenAI-compatible server with continuous batching for MLX on Apple Silicon

LLM inference server with continuous batching, TTFT/throughput metrics, and Prometheus/Grafana monitoring