Search

A high-throughput and memory-efficient inference and serving engine for LLMs

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

SGLang is a high-performance serving framework for large language models and multimodal models.

Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

A file server that supports static serving, uploading, searching, accessing control, webdav...

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

A Datacenter Scale Distributed Inference Serving Framework

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.

Apache Pinot: a real-time analytics database for low-latency SQL and vector search, serving user-facing and agent-facing applications

A node module for serving static files. Does etags, caching, etc.

A flexible, high-performance serving system for machine learning models

A framework for serving GraphQL from Laravel

Open source platform for X.509 certificate based service authentication and fine grained access control in dynamic infrastructures. Athenz supports provisioning and configuration (centralized authorization) use cases as well as serving/runtime (decentralized authorization) use cases.

Repository of sample applications for https://vespa.ai, the open big data serving engine

Plugin for serving static files as fast as possible

Kubernetes-based, scale-to-zero, request-driven compute

Helper application for Linux distributions serving as a kind of "entry point" for running and integrating AppImages

A distributed approximate nearest neighborhood search (ANN) library which provides a high quality vector index build, search and distributed online serving toolkits for large scale vector search scenario.

A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution

A cross-platform, high-performance and asynchronous web server for static files-serving. ⚡

API server for serving backend requests for trace.moe

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

Nintendo Switch library manager, with automated management tasks, serving your library to multiple supported clients on your Switch, with shop customization and multi user authentication.

A simple and ultrafast http reverse proxy serving multiple domain names and terminating TLS for http/1.1, 2 and 3, written in Rust

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

The Runpod worker template for serving our large language model endpoints. Powered by vLLM.