huggingface/tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Rust10,760 stars1,102 forksApache-2.0

bertgptlanguage-modelnatural-language-processingnatural-language-understandingnlptransformers

Rankings

Daily

#3866

Weekly

#2948

Monthly

#2166

Total stars

Daily new stars

Get badge

This repository is tracked by Trending Repos. The badge upgrades automatically if it ever cracks the top 100.

Markdown

![Trending Repos](https://trending-repos.com/badge/huggingface/tokenizers.svg)

HTML

<img src="https://trending-repos.com/badge/huggingface/tokenizers.svg" alt="Trending Repos" />

Raw URL

https://trending-repos.com/badge/huggingface/tokenizers.svg