MachineLearning · Natural Language Processing

Natural Language Processing

• huggingface/tokenizers on GitHub, opens in a new tab

  11Kstars

  Fast State-of-the-Art Tokenizers optimized for Research and Production

  Rust·Apache License 2.0·active·9 hours ago

• Michael-A-Kuykendall/shimmy on GitHub, opens in a new tab

  5.8Kstars

  Python-free Rust inference server for NLP models with OpenAI API compatibility and hot model swapping.

  Rust·Apache License 2.0·active·2 weeks ago

• guillaume-be/rust-bert on GitHub, opens in a new tab

  3.1Kstars

  Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)

  Rust·Apache License 2.0·steady·7 months ago

• Michael-A-Kuykendall/shimmytok on GitHub, opens in a new tab

  23stars

  Pure-Rust tokenizer for GGUF models, compatible with llama.cpp tokenization.

  Rust·Apache License 2.0·active·2 weeks ago