Hugging Face Tokenizers
Ultra-fast tokenizer library (Rust core) — BPE, WordPiece, SentencePiece — tokenize GBs in seconds.
Hugging Face Tokenizers
Tokenizers provides an extremely fast and versatile tokenization library. The core is written in Rust for maximum performance: it can tokenize a gigabyte of text in under 20 seconds on a single CPU. All modern tokenizer algorithms are supported, with full alignment tracking and pre/post-processing pipelines.
Key Features
- Rust-powered speed: 1 GB/s+ tokenization throughput
- All major algorithms: BPE (GPT-2), WordPiece (BERT), Unigram (SentencePiece), WordLevel
- Alignment tracking: maps token positions back to original characters
- Full pre/post-processing: normalizers, pre-tokenizers, post-processors, decoders
- Train from scratch or load pretrained tokenizers from the Hub
- Python, Node.js, and Rust APIs
Quick Start
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
# Train a new tokenizer
tokenizer = Tokenizer(BPE())
trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]"])
tokenizer.train(["data.txt"], trainer)
# Or load a pretrained one
tokenizer = Tokenizer.from_pretrained("bert-base-uncased")
encoding = tokenizer.encode("Hello world!")
print(encoding.tokens) # ['[CLS]', 'hello', 'world', '!', '[SEP]']
Install via ai-supply
npx ai-supply add huggingface-tokenizers-fast
Curated mirror of the open-source Hugging Face Tokenizers (Apache-2.0). Get it from the source.
Compromise signals — malicious or tampered code (leaked secrets, backdoors, a dropped executable) — reduce the score, and known dependency CVEs carry a bounded penalty (they warrant review but never QUARANTINE — update the dependency to clear). Other dangerous-by-capability traits are risk surface, expected for some capabilities. Every finding is mapped to its OWASP control below.
Findings mapped to the OWASP Top 10 for LLM Applications (2025) and the OWASP Machine Learning Security Top 10. Expand any flagged control for the exact findings — compromise reduces the score; expected/risk-surface do not, except a known CVE, which carries a small bounded penalty (high/critical → Review).
The same gate an agent runs before installing (POST /api/v1/trust/huggingface-tokenizers-fast/check). Click a policy:
Consume Hugging Face Tokenizers programmatically. Authenticate with an API key or session — see Authorize an agent.
# Agents: CHECK BEFORE YOU INSTALL (no auth) — score, grade, level, capability manifest
curl https://ai-supply.store/api/v1/trust/huggingface-tokenizers-fast
# Gate against your org policy (returns { pass, violations })
curl -X POST https://ai-supply.store/api/v1/trust/huggingface-tokenizers-fast/check \
-H "Content-Type: application/json" \
-d '{"minGrade":"B","denyPermissions":["shell"],"denyUnknownEgress":true}'
# CLI
npx ai-supply add huggingface-tokenizers-fast
# REST (install → download)
curl -X POST https://ai-supply.store/api/v1/listings/huggingface-tokenizers-fast/install \
-H "Authorization: Bearer $AIM_KEY"
# MCP tool
install_listing({ "slug": "huggingface-tokenizers-fast" })OpenAPI spec →Curated mirror — latest upstream source. See the repository for tagged releases.