Find trending repositories by name or description.
Open source, composable payments platform | PCI compliant | SaaS and Self-host options | Enables connectivity to multiple payment, payout, fraud, vault and tokenization providers | Uplifts authorization with intelligent routing and revenue recovery | Reduce payment processing costs with cost observability | Reduces payment ops with reconciliation
CoreNLP: A Java suite of core NLP tools for tokenization, sentence segmentation, NER, parsing, coreference, sentiment analysis, etc.
:book: A Golang library for text processing, including tokenization, part-of-speech tagging, and named-entity extraction.
Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages
Language model tokenization at GB/s
Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across vLLM, TRT-LLM, TokenSpeed, SGLang, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.
[ICLR 2026] LiTo: Surface Light Field Tokenization
Learn LLM internals step by step - from tokenization to attention to inference optimization.
ICML2026: Resolving Manifold Misalignment in Visual Tokenization via Topological Orthogonality