Hugging Face Tokenizers

Hugging Face Tokenizers

Tokenizers provides an extremely fast and versatile tokenization library. The core is written in Rust for maximum performance: it can tokenize a gigabyte of text in under 20 seconds on a single CPU. All modern tokenizer algorithms are supported, with full alignment tracking and pre/post-processing pipelines.

Key Features

• Rust-powered speed: 1 GB/s+ tokenization throughput
• All major algorithms: BPE (GPT-2), WordPiece (BERT), Unigram (SentencePiece), WordLevel
• Alignment tracking: maps token positions back to original characters
• Full pre/post-processing: normalizers, pre-tokenizers, post-processors, decoders
• Train from scratch or load pretrained tokenizers from the Hub
• Python, Node.js, and Rust APIs

Quick Start

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer

# Train a new tokenizer
tokenizer = Tokenizer(BPE())
trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]"])
tokenizer.train(["data.txt"], trainer)

# Or load a pretrained one
tokenizer = Tokenizer.from_pretrained("bert-base-uncased")
encoding = tokenizer.encode("Hello world!")
print(encoding.tokens)  # ['[CLS]', 'hello', 'world', '!', '[SEP]']

Install via ai-supply

npx ai-supply add huggingface-tokenizers-fast

Curated mirror of the open-source Hugging Face Tokenizers (Apache-2.0). Get it from the source.