When the network blinks
You send a request to an API. The server drops the packet. Your code receives an error and crashes. The user sees a blank screen. This happens because networks are unreliable. Packets get lost. Servers time out. DNS resolves slowly. A single failure shouldn't kill your application. You need a way to try again.
Retry logic is the practice of repeating a failed operation until it succeeds or a limit is reached. It turns transient failures into temporary delays. Without retries, your software is fragile. With retries, it becomes resilient. The challenge is implementing retries without overwhelming the server or wasting your own resources.
The concept: patience with limits
Think of retry logic like calling a friend who isn't answering. You call once. No answer. You call again immediately. Still no answer. If you keep calling every second, you annoy your friend and drain your battery. A better approach is to wait a bit longer between attempts. You call, wait five seconds, call again, wait ten seconds, call again. Eventually, they pick up, or you give up and send a text instead.
In Rust, this pattern uses a loop, a counter, and a delay. The loop keeps trying. The counter ensures you stop eventually. The delay prevents you from hammering the server. You also need to decide which errors are worth retrying. A timeout is worth retrying. A "not found" error is not. The server won't magically create the missing page if you ask again.
Minimal example: the naive loop
The simplest retry implementation uses a loop and a manual counter. You attempt the request. If it fails, you increment the counter and sleep. If the counter hits the limit, you give up and return the error.
use std::time::Duration;
use std::thread;
/// Attempts to fetch data, retrying on failure up to max_retries times.
fn fetch_with_retry(url: &str, max_retries: u32) -> Result<String, String> {
let mut attempts = 0;
// WHY: loop runs indefinitely until a return statement is hit.
loop {
match fetch(url) {
// WHY: Return immediately on success.
Ok(data) => return Ok(data),
Err(e) => {
attempts += 1;
// WHY: Check the limit before sleeping to avoid wasted delay on the final failure.
if attempts >= max_retries {
return Err(e);
}
// WHY: Pause to avoid hammering the server.
thread::sleep(Duration::from_secs(1));
}
}
}
}
fn fetch(url: &str) -> Result<String, String> {
// Simulated network call
Err("Network error".to_string())
}
This code works. It retries. It stops. It sleeps. It is also naive. The sleep duration is fixed. The error type is a string. The implementation blocks the current thread. For a quick script, this is fine. For a real application, you need more sophistication.
Don't ship fixed-delay retries to production. They scale poorly and annoy upstream services.
Why naive retries fail
Fixed delays create problems under load. Imagine ten clients all fail at the same time. They all wait one second. They all retry at the same time. The server gets hit by a wave of requests. This is the thundering herd problem. The server might be recovering, and the synchronized retries push it back into failure.
You also waste time. If the server is down for five minutes, retrying every second for two hundred times adds no value. You want to back off. You want to wait longer between attempts as failures accumulate. This gives the server time to recover and spreads your load over time.
The standard solution is exponential backoff. You wait one second, then two, then four, then eight. The delay grows exponentially. This naturally spaces out retries and reduces pressure on the server.
Realistic example: exponential backoff with jitter
A production-ready retry strategy combines exponential backoff with jitter. Exponential backoff increases the wait time. Jitter adds randomness. Randomness prevents clients from synchronizing their retries. If everyone adds a random amount to their delay, the thundering herd dissolves into a gentle rain of requests.
You also need better error handling. String errors don't tell you why the request failed. You need an error type that distinguishes between retryable errors (timeouts, connection refused) and fatal errors (404, authentication failure). Retrying a fatal error is a waste of time.
use std::time::Duration;
use std::thread;
use std::collections::HashMap;
/// Represents the different ways a request can fail.
#[derive(Debug)]
enum RequestError {
/// The server is temporarily unavailable. Safe to retry.
Transient(String),
/// The resource does not exist. Retrying will never help.
NotFound,
/// Authentication failed. Retrying will never help.
AuthFailed,
}
/// Calculates the delay for the current attempt using exponential backoff with jitter.
///
/// # Arguments
/// * `attempt` - The current attempt number (0-indexed).
/// * `max_delay` - The maximum delay in seconds to prevent waiting forever.
///
/// # Returns
/// A Duration representing how long to wait.
fn calculate_backoff(attempt: u32, max_delay: Duration) -> Duration {
// WHY: Exponential growth: 2^0, 2^1, 2^2...
let exponential = 2u64.pow(attempt);
// WHY: Convert to milliseconds for jitter calculation.
let base_ms = exponential * 1000;
// WHY: Add random jitter between 0 and base_ms to desynchronize clients.
// In a real app, use rand::thread_rng().gen_range(0..base_ms).
// Here we simulate jitter with a simple modulo for demonstration.
let jitter = (attempt * 123) % base_ms;
let total_ms = base_ms + jitter;
// WHY: Cap the delay so we don't wait hours on a failed request.
let capped_ms = total_ms.min(max_delay.as_millis() as u64);
Duration::from_millis(capped_ms)
}
/// Fetches data with exponential backoff and jitter.
fn fetch_with_backoff(url: &str, max_retries: u32) -> Result<String, RequestError> {
let max_delay = Duration::from_secs(60);
let mut attempt = 0;
loop {
match fetch_realistic(url) {
Ok(data) => return Ok(data),
Err(RequestError::NotFound) | Err(RequestError::AuthFailed) => {
// WHY: Fatal errors should fail immediately. No point retrying.
return Err(e);
}
Err(e) => {
if attempt >= max_retries {
return Err(e);
}
let delay = calculate_backoff(attempt, max_delay);
// WHY: Log the retry attempt for observability.
eprintln!("Attempt {} failed. Retrying in {:?}", attempt + 1, delay);
thread::sleep(delay);
attempt += 1;
}
}
}
}
fn fetch_realistic(url: &str) -> Result<String, RequestError> {
// Simulated network call that returns specific error types
Err(RequestError::Transient("Connection timeout".to_string()))
}
This implementation is robust. It backs off exponentially. It adds jitter. It caps the delay. It distinguishes between retryable and fatal errors. It logs its actions. This is the pattern you should follow for synchronous code.
Jitter is not optional. Without it, your retries will synchronize and cause outages.
The async reality
Network requests in Rust are almost always asynchronous. Using std::thread::sleep blocks the entire thread. If you run this in a web server, you block a worker thread. If all threads block on retries, your server stops responding to new requests. You need async sleep.
The logic remains the same. The loop becomes an async loop. The sleep becomes tokio::time::sleep. The error handling stays identical. The key difference is that your thread is free to do other work while waiting for the delay to expire.
use tokio::time::{sleep, Duration};
/// Async version of fetch with exponential backoff.
async fn fetch_with_backoff_async(url: &str, max_retries: u32) -> Result<String, RequestError> {
let max_delay = Duration::from_secs(60);
let mut attempt = 0;
loop {
match fetch_async(url).await {
Ok(data) => return Ok(data),
Err(RequestError::NotFound) | Err(RequestError::AuthFailed) => {
return Err(e);
}
Err(e) => {
if attempt >= max_retries {
return Err(e);
}
let delay = calculate_backoff(attempt, max_delay);
// WHY: Async sleep yields the thread. Other tasks can run.
sleep(delay).await;
attempt += 1;
}
}
}
}
async fn fetch_async(url: &str) -> Result<String, RequestError> {
// Simulated async network call
Err(RequestError::Transient("Async timeout".to_string()))
}
Async retry logic is the standard for modern Rust applications. It scales. It doesn't block. It integrates with the rest of the async ecosystem.
Blocking sleep in an async context is a trap. Use tokio::time::sleep or the equivalent for your runtime.
Pitfalls and compiler errors
Retry logic introduces specific pitfalls. The compiler will catch some. Others will bite you at runtime.
Infinite loops: If you forget to increment the counter or check the limit, your loop runs forever. The compiler won't stop you. You must manage the state yourself.
Moved values: If you move a value into the request function, you can't use it again in the loop. The compiler rejects this with E0382 (use of moved value). You need to clone the data or restructure the code so the value is created inside the loop.
// BAD: data is moved into fetch on the first iteration.
let data = prepare_data();
loop {
match fetch(&data) { ... } // Error on second iteration: data is moved.
}
// GOOD: Prepare data inside the loop or clone it.
loop {
let data = prepare_data();
match fetch(&data) { ... }
}
Mutable borrows: If you try to mutate a variable while it's borrowed, the compiler rejects you with E0502 (cannot borrow as mutable because it is also borrowed as immutable). This often happens if you hold a reference to the result while trying to update the retry state. Ensure borrows end before you mutate.
Retrying everything: Retrying a 404 error is useless. The server told you the resource doesn't exist. Asking again won't change that. You must classify errors. If your error type is just a string, you can't classify. You need an enum or a structured error type.
Ignoring jitter: As mentioned, synchronized retries cause thundering herds. Always add jitter. Even a simple random number generator helps.
Treat the error type as the gatekeeper. If the error isn't retryable, fail fast.
Decision: when to use this vs alternatives
Retry logic varies in complexity. Choose the approach that matches your needs.
Use a manual loop with fixed delay when you are writing a quick script or a one-off tool where simplicity matters more than resilience. Use exponential backoff with jitter when you are building a client that interacts with external APIs or services that may experience transient failures. Use a dedicated library like retry or reqwest-middleware when you need advanced features like circuit breakers, rate limiting, or complex retry policies without reinventing the wheel. Use async retry logic when your application is built on an async runtime like tokio or async-std to avoid blocking threads. Use immediate failure for errors that are never transient, such as authentication failures or missing resources.
Trust the backoff curve. It protects your server and yours.