Search

A high-throughput and memory-efficient inference and serving engine for LLMs

The fastest, litest AI Gateway. Rust core with Python SDK. Call 100+ LLM APIs in OpenAI (or native) format with cost tracking, guardrails, load balancing, and logging [Bedrock, Azure, OpenAI, Anthropic, OpenAI, VertexAI, vLLM, Nvidia NIM]

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

Community maintained hardware plugin for vLLM on Huawei Ascend

State-of-the-art LLM compression, built for production inference with vLLM

a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features (GGUF, RadixAttention, Cache-aware scheduling, ...)

Fully uncensored, capability-enhanced abliteration of Qwen3.6-27B. NVFP4 + z-lab DFlash speculative decoding (n=12) on the unified ghcr.io/aeon-7/aeon-vllm-ultimate:latest container, tuned for long-context draft acceptance on DGX Spark. 6 HF variants (BF16/NVFP4/MTP/MTP-XS), docker-compose, and QuickStart.

Moving to vllm-project/vllm-jax. TPU inference for vLLM, with unified JAX and PyTorch support.

Community maintained hardware plugin for vLLM on MetaX GPU

Sophon-Realtime v1: open full-duplex voice models that hear your tone and speak with one. Tag and Flash run on one GPU with vLLM-Omni; E2E in preview.

DeepSeek-V4.1-Flash on 2x NVIDIA DGX Spark with the TensorFold engine: 1.8-1.9x code, 2.5x multi-stream, 1.7-1.95x prefill vs the vLLM kit, exact speculative decoding. Work in progress.

Reproducible Docker/vLLM deployment, tuning, and benchmarks for DeepSeek V4.1 Flash on a switchless three-node or four-node DGX Spark fabric.

The Runpod worker template for serving our large language model endpoints. Powered by vLLM.

An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)

Efficent platform for inference and serving local LLMs including an OpenAI compatible API server.

Fully-featured, beautiful web interface for vLLM - built with NextJS.

A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

Cost-efficient and pluggable Infrastructure components for GenAI inference

Reliable model swapping for any local OpenAI/Anthropic compatible server - llama.cpp, vllm, etc

vLLM’s reference system for K8S-native cluster-wide deployment with community-driven performance optimization

Lumina-Image 2.0: A Unified and Efficient Image Generative Framework

ArcticInference: vLLM plugin for high-throughput, low-latency inference

A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM

A lightweight, configurable, and real-time simulator designed to mimic the behavior of vLLM without the need for GPUs or running actual heavy models.

learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

Common recipes to run vLLM

A high-performance and light-weight router for vLLM large scale deployment

AI Client for chat, RAG, Skills, MCP tools, and agents. Support multiple LLMs (Anthropic, OpenAI, VertexAI, vLLM, Nvidia NIM, Gemini, OpenRouter)

A programmable Mixture-of-Models router for heterogeneous LLM inference