How to Use Tokenizers in Rust
You're building a text classifier. You have a user review: "This product is amazing!" You need to feed that into a neural network. Neural networks don't read English. They eat vectors of numbers. You need a translator that chops your text into chunks, maps those chunks to IDs, and hands you a sequence of integers. That translator is a tokenizer. In Rust, the tokenizers crate handles this heavy lifting, giving you the same speed and flexibility as the Python ecosystem without the GIL or the overhead.
What a tokenizer actually does
Tokenization is the process of breaking text into smaller units called tokens. These tokens can be words, subwords, or characters, depending on how the tokenizer was trained. The tokenizer holds a vocabulary: a map from text strings to integer IDs. When you encode text, the tokenizer looks up each token in the vocabulary and returns the corresponding ID. If a word isn't in the vocabulary, the tokenizer uses a strategy like breaking it into subwords or marking it as unknown.
Think of a tokenizer like a specialized chef's knife combined with a recipe book. The recipe book (the model file) tells the chef exactly where to cut. "Pizza" might be one chunk. "Pepperoni" might be two chunks if the book doesn't know the word. The knife chops the text according to the book. The result is a list of ingredient IDs the kitchen (the model) understands.
The tokenizers crate implements common algorithms like BPE (Byte-Pair Encoding), WordPiece, and Unigram. It reads model files (usually JSON) generated by training scripts and applies them to your input. Tokenization turns text into math the model can digest.
Minimal example
Add the crate to your dependencies. The version 0.19 is current and stable.
[dependencies]
tokenizers = "0.19"
Load a tokenizer from a file and encode a string. The tokenizer.json file contains the vocabulary and the algorithm configuration. This file is usually provided alongside the model weights.
use tokenizers::Tokenizer;
/// Load a tokenizer and encode a simple string.
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Load the tokenizer model from a JSON file.
// This parses the vocabulary and algorithm settings into memory.
let tokenizer = Tokenizer::from_file("tokenizer.json")?;
// Encode a string into tokens.
// The second argument controls whether to add special tokens like [CLS] or [SEP].
// Set this to true for models like BERT that expect special tokens.
let encoding = tokenizer.encode("Hello, world!", false)?;
// Extract the integer IDs for each token.
// These are the numbers you feed into the model.
let ids = encoding.get_ids();
println!("Token IDs: {:?}", ids);
// Get the actual text strings for each token.
// Useful for debugging or logging.
let tokens = encoding.get_tokens();
println!("Tokens: {:?}", tokens);
Ok(())
}
Load the model once. Encode text as many times as you need.
Walkthrough
When you call Tokenizer::from_file, the crate reads the JSON file and constructs an in-memory representation of the vocabulary and the tokenization algorithm. This step happens once. You can reuse the same Tokenizer instance across many calls. The encode method takes a string and runs the tokenization algorithm. It splits the text, normalizes it, and maps tokens to IDs.
Normalization happens before splitting. The tokenizer applies rules defined in the model file, such as lowercasing, stripping accents, or composing Unicode characters. This ensures consistent tokenization across different input formats. If your model was trained with lowercasing, the tokenizer applies it automatically. Disabling normalization can break the model's performance.
The result is an Encoding object. This object holds the IDs, the original tokens, and offsets pointing back to the original string. You can extract IDs with get_ids, tokens with get_tokens, and character spans with get_offsets. The Encoding object is efficient. It avoids copying data where possible.
Realistic workflow: padding, truncation, and decoding
Real models have constraints. They expect sequences of a fixed length. They might reject inputs that are too long. You need to pad short sequences and truncate long ones. The tokenizers crate provides configuration objects for these operations.
use tokenizers::{Tokenizer, TruncationParams, PaddingParams};
/// Configure and use a tokenizer with padding and truncation.
fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut tokenizer = Tokenizer::from_file("tokenizer.json")?;
// Configure truncation to cut text at 128 tokens.
// This prevents feeding sequences that are too long for the model.
// LongestFirst strategy truncates the longest input in a batch first.
let truncation = TruncationParams {
max_length: 128,
stride: 0,
strategy: tokenizers::TruncationStrategy::LongestFirst,
direction: tokenizers::TruncationDirection::Right,
};
tokenizer.with_truncation(Some(truncation));
// Configure padding to fill sequences to 128 tokens.
// Models often require fixed-size inputs for batching.
// The pad_id and pad_token must match the model's expectations.
let padding = PaddingParams {
length: Some(128),
stride: 0,
pad_to_multiple_of: None,
pad_id: 0,
pad_type_id: 0,
pad_token: "[PAD]".to_string(),
};
tokenizer.with_padding(Some(padding));
// Encode with padding and truncation applied automatically.
// The encoding is now exactly 128 tokens long.
let encoding = tokenizer.encode("This is a short text.", true)?;
assert_eq!(encoding.len(), 128);
// Decode the IDs back to text.
// skip_special_tokens removes markers like [CLS] and [PAD].
let decoded = tokenizer.decode(&encoding.get_ids(), true)?;
println!("Decoded: {}", decoded);
Ok(())
}
Set your padding and truncation rules upfront. The tokenizer applies them automatically on every call.
Offsets and spans
Sometimes you need to map tokens back to the original text. This is useful for highlighting entities in a UI or extracting context. The Encoding object provides byte offsets for each token.
use tokenizers::Tokenizer;
/// Map tokens back to the original text using offsets.
fn main() -> Result<(), Box<dyn std::error::Error>> {
let tokenizer = Tokenizer::from_file("tokenizer.json")?;
let original_text = "The quick brown fox.";
let encoding = tokenizer.encode(original_text, false)?;
// Iterate over tokens and their offsets.
// Offsets are byte indices into the original string.
for (token, offset) in encoding.get_tokens().iter().zip(encoding.get_offsets().iter()) {
let start = offset.start;
let end = offset.end;
// Safety: Offsets are guaranteed to be valid byte boundaries
// within the original string by the tokenizer implementation.
let original_slice = &original_text[start..end];
println!("Token '{}' maps to '{}'", token, original_slice);
}
Ok(())
}
Use offsets to bridge the gap between model outputs and human-readable text.
Pitfalls and errors
If you forget to enable special tokens when the model expects them, your sequence will be missing the start and end markers. The model might output nonsense or crash. The encode method takes a boolean for this. Set it to true for models like BERT. If you load a tokenizer file that doesn't exist, from_file returns an error. You must handle this with ? or match. The compiler won't save you from a missing file; that's a runtime error.
If you pass a &str where a &[u32] is expected, the compiler rejects the code with E0308 (mismatched types). Check your method signatures. If you try to decode IDs that aren't in the vocabulary, the tokenizer might return unknown tokens or panic depending on the configuration. Validate your IDs before decoding.
Read the model card. The tokenizer settings are part of the model definition.
When to use tokenizers
Use tokenizers when you are working with transformer models from Hugging Face. The crate reads the standard tokenizer.json format and supports BPE, WordPiece, Unigram, and other algorithms used by modern LLMs. Use tokenizers when you need production-grade performance. The crate is written in Rust and C++, offering speed comparable to Python implementations without the overhead. Use tokenizers when you need advanced features like padding, truncation, and offset mapping. The API provides configuration objects for these operations, making it easy to prepare batches for inference.
Reach for simple string splitting when your task only requires breaking text by whitespace or punctuation. If you don't have a vocabulary and don't need subword tokenization, str::split is sufficient. Reach for unicode-segmentation when you need to split text by graphemes or words according to Unicode standards. This crate handles complex scripts and combining characters correctly without a learned model.
Match the tool to the vocabulary. No vocab means no tokenizers.