The problem with hoping callers remember
You are building a database client. You write a query method that sends SQL to a server. You also write a connect method that establishes the network link. Everything looks good. You publish the crate. A user downloads it, creates a client, and immediately calls query. The app crashes with a panic: "connection not established."
You could return a Result from query and check the connection state inside. That works, but the error happens at runtime. The user sees a crash or an error message. You could add a connected boolean and check it, but that costs CPU cycles on every call, and the user can still call query in the wrong order. You want the compiler to reject the code if the user tries to query before connecting. You want the error to appear before the binary is built.
That is what the token pattern solves. It uses a unique type that the caller cannot construct to prove that a precondition is met. The caller must acquire the token from a trusted source by performing the correct setup. Once they have the token, they can call the protected function. If they skip the setup, they have no token, and the code won't compile. The token acts as a compile-time proof of state.
The token as a compile-time proof
Think of a token like a physical key to a secure room. You cannot just walk in. You need the key. The key proves you went through security and were granted access. In Rust, a token is a type that lives inside your module. It has no public constructor. Callers can see the type, but they cannot create an instance of it.
The only way to get a token is to call a function that returns one. That function is the trusted source. It checks the state, performs the setup, and hands back the token. The compiler trusts the token. If a function requires a token as an argument, the compiler knows the caller must have called the trusted source first. The chain of custody is enforced by the type system.
This pattern eliminates entire classes of logic errors. You cannot call a function in the wrong state because you literally cannot produce the argument required to call it. The error shifts from runtime panics to compile-time type errors. The user gets a clear message from the compiler instead of a crash in production.
Minimal example: a file that must be opened
Here is a simple implementation. We have a SafeFile struct. It has a write method, but write requires a WriteToken. The token can only be obtained by calling open.
// The token type. Defined in this module with no public constructor.
// Callers cannot create this type outside this module.
struct WriteToken;
struct SafeFile {
// Internal state. Hidden from callers.
_inner: (),
}
impl SafeFile {
pub fn new() -> Self {
SafeFile { _inner: () }
}
/// Opens the file. Returns a token if successful.
/// This is the trusted source. Only here can we create a WriteToken.
pub fn open(&mut self) -> Option<WriteToken> {
// Simulate file opening logic.
// If this succeeds, we return the token.
Some(WriteToken)
}
/// Writes data. Requires a WriteToken.
/// The token proves that open() was called and succeeded.
pub fn write(&mut self, _token: WriteToken, data: &[u8]) -> std::io::Result<()> {
// We can safely assume the file is open.
// The compiler guaranteed we received a token from open().
println!("Writing data...");
Ok(())
}
}
fn main() {
let mut file = SafeFile::new();
// This compiles: we get the token from the trusted source.
if let Some(token) = file.open() {
let _ = file.write(token, b"Hello, safe world!");
}
// This fails to compile: we cannot construct WriteToken manually.
// The compiler rejects this with E0603 (private struct) or a similar error.
// let fake_token = WriteToken;
// file.write(fake_token, b"Bad data");
}
The token is a unit struct. It has zero size. It takes no memory. The compiler optimizes it away completely. There is zero runtime overhead. The safety comes entirely from the type system.
How the compiler enforces the flow
The magic relies on privacy. WriteToken is defined in the module. It has no fields, or its fields are private. The caller sees the type name, but they cannot build it. If they try WriteToken {}, the compiler rejects the code. The error is usually E0603 (struct WriteToken is private) or a message about private fields.
The caller is stuck. They need a WriteToken to call write, but they cannot make one. The only door is open. When open returns a WriteToken, the compiler accepts it. You, the library author, are the only one who can mint tokens. When you return a token, you are making a promise to the compiler: "The state is valid. The file is open. The user can proceed."
The compiler treats the token as a certificate. It doesn't check the internal state again. It trusts the token. This is why the token pattern is a zero-cost abstraction. The runtime check is replaced by a compile-time proof. The function write doesn't need to check if the file is open. It just proceeds. The cost of the check is paid once, at compile time, by the caller having to acquire the token.
Realistic example: a database client
In real code, tokens are often used for configuration or connection management. Here is a database client that requires a connection token before queries are allowed.
pub struct Database {
// Internal connection state.
_inner: (),
}
// The token. Callers cannot create this.
struct ConnectedToken;
impl Database {
pub fn new() -> Self {
Database { _inner: () }
}
/// Connects to the database.
/// Returns a ConnectedToken if the connection succeeds.
pub fn connect(&mut self) -> Result<ConnectedToken, String> {
// Simulate connection logic: handshake, auth, etc.
// If this fails, we return an error and no token.
// The caller cannot proceed without handling the error.
Ok(ConnectedToken)
}
/// Executes a query.
/// Requires a ConnectedToken. The token proves we are connected.
pub fn execute(&self, _token: ConnectedToken, query: &str) -> Result<String, String> {
// No runtime check needed. The token is the proof.
// We can assume the connection is valid.
Ok(format!("Result for: {}", query))
}
}
fn main() {
let mut db = Database::new();
// Must connect first. The Result forces error handling.
let token = db.connect().expect("Failed to connect");
// Now we can execute queries.
let _ = db.execute(token, "SELECT * FROM users");
// This won't compile: no token available.
// db.execute(?, "DROP TABLE users");
}
Returning the token inside a Result is a common pattern. If the setup fails, the caller gets an error and no token. They cannot call the protected function. If the setup succeeds, they get the token and can proceed. This forces the caller to handle errors explicitly. They cannot ignore the connection failure and accidentally call execute.
Convention: naming tokens and handling clones
Naming matters. The token name should signal what capability or state it represents. WriteToken is better than Token. ConnectedToken is better than AuthKey. ConfiguredToken is better than BuilderToken. The name tells the caller what they are proving. If the name is vague, the API is harder to understand.
Another convention is to decide whether the token should be cloneable. If the token represents a unique capability, do not derive Clone. If the user can clone the token, they can duplicate the capability. This might be safe, or it might be a bug. If the token proves a file is open, cloning it is usually fine. If the token proves a transaction is active, cloning might allow multiple commits, which could be dangerous. The default is to leave tokens as non-cloneable unless duplication is explicitly safe. If you need to pass the token to multiple places, consider returning multiple tokens or using a reference.
Owned versus borrowed tokens
Tokens can be owned or borrowed. An owned token like ConnectedToken can be moved around. The caller holds the token. A borrowed token like &ConnectedToken ties the token to a lifetime. The token cannot outlive the object that created it.
Use an owned token when the caller needs to store the token or pass it across threads. Use a borrowed token when the token should die when the object dies. Borrowed tokens are useful for enforcing that the token is only valid while the object exists. For example, a WriteGuard that borrows the file handle. If the file is dropped, the guard becomes invalid. The compiler enforces this via lifetimes.
struct WriteGuard<'a> {
_file: &'a mut File,
}
impl<'a> WriteGuard<'a> {
fn write(&mut self, data: &[u8]) {
// Access self._file safely.
}
}
This is a variation of the token pattern where the token holds a reference. It combines the token proof with lifetime safety. The token proves the file is open, and the lifetime proves the file exists.
Consuming tokens for single-use actions
Sometimes a token should be used only once. You can enforce this by consuming the token. The function takes ownership of the token and does not return it. The caller cannot reuse the token after the call.
struct CommitToken;
impl Database {
pub fn start_transaction(&mut self) -> CommitToken {
CommitToken
}
/// Commits the transaction. Consumes the token.
/// The token cannot be used again.
pub fn commit(&mut self, _token: CommitToken) -> Result<(), String> {
Ok(())
}
}
If the caller tries to call commit twice, they will fail. They only have one token. After the first call, the token is gone. This is useful for actions that should happen exactly once, like committing a transaction or finalizing a build. The compiler enforces single-use.
Pitfalls: broken promises and leakage
The token pattern relies on a promise. When you return a token, you promise that the state is valid. If you return a token but the state is wrong, you have introduced a logic bug. The compiler cannot catch this. It trusts the token. If you break the promise, the API is broken. The caller will get a runtime error or undefined behavior, even though the code compiled.
Always verify the state before returning a token. If the setup fails, do not return a token. Return an error or None. The token is a certificate of validity. Do not issue certificates for invalid states.
Another pitfall is token leakage. If the token type is public and has public fields, callers might be able to construct it. Ensure the token is private or has no public constructor. A unit struct with no fields is the safest choice. It cannot be constructed outside the module. If you need to store data in the token, make the fields private.
If you try to construct a private token outside the module, the compiler rejects it with E0603 (private struct) or a similar privacy error. This is the compiler protecting your invariant. Trust it.
Decision: tokens versus alternatives
Use the token pattern when you need to enforce a precondition at compile time and want zero runtime overhead. Use the token pattern when you want to hide internal state while proving a capability exists. Use the token pattern when you want to force the caller to perform a setup step before accessing a resource.
Reach for the typestate pattern when the state transitions are complex and you want the type signature to change as the object evolves. The typestate pattern encodes the state in the type itself, so the available methods change based on the state. This is more expressive but heavier to implement.
Reach for Result when you need to report errors but don't need to prevent the function call itself. If the caller can handle the error and proceed, Result is sufficient. Tokens are for preventing the call entirely.
Pick a simple flag or boolean when the safety requirement is low and you value API flexibility over strict enforcement. If a mistake is harmless or easy to recover from, tokens might be overkill.
Treat the token as a contract. If you return it, the state must be valid. The compiler trusts you. Don't break that trust.