Search

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features (GGUF, RadixAttention, Cache-aware scheduling, ...)

DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

Supercharged batching to save you fees and preserve your privacy

A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.

RapidLLM is an LLM inference framework with continuous batching, tensor and data parallelism, quantized weights, and with pluggable Triton/CUDA kernels.

High-performance LLM inference engine — drop-in replacement for Ollama with faster multi-turn inference, lower TTFT, and higher throughput through prefix caching and continuous batching.

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

Pluggable data replication with checkpointing, batching, and integrity events