The blind spot in production
You deploy a Rust web service. It passes every test. In production, a request hangs for three seconds before returning a 500 error. The logs show nothing useful. The database query took two seconds, but you have no idea which handler triggered it, which upstream service called yours, or where the time actually went. You are debugging a black box.
How distributed tracing actually works
OpenTelemetry turns that black box into a transparent pipeline. Think of it like tracking a package through a global shipping network. The package is the HTTP request. Each warehouse it passes through is a function, a database call, or an external API. The tracking number is the trace ID. Every time the package moves to a new station, a worker scans it and logs the timestamp. When you look at the final report, you see exactly where it sat idle, which station caused the delay, and how long each step took.
In Rust, the package is your request. The stations are your handlers and async tasks. The tracking system is a combination of the tracing crate and the OpenTelemetry protocol. tracing gives you a lightweight, zero-cost way to mark the start and end of work. OpenTelemetry translates those marks into a standard format that backends like Jaeger, Zipkin, or Datadog can ingest. The bridge between them is the tracing-opentelemetry layer. It intercepts your tracing spans, converts them to OpenTelemetry events, and ships them out.
Context propagation is the invisible glue. When a request crosses a thread boundary or jumps between async tasks, the trace ID must travel with it. Rust's async runtime does not automatically copy tracing context. You have to tell it to. The TraceContext propagator handles this by reading standard HTTP headers, attaching the trace ID to the current execution context, and ensuring every child span inherits the parent's tracking number.
A minimal pipeline you can run today
You do not need a full observability backend to verify the wiring. The opentelemetry-stdout crate prints traces directly to your terminal. This proves the pipeline is functional before you commit to a production exporter.
[dependencies]
tracing = "0.1"
tracing-subscriber = "0.3"
tracing-opentelemetry = "0.23"
opentelemetry = "0.22"
opentelemetry-stdout = "0.3"
tokio = { version = "1", features = ["full"] }
use opentelemetry::global;
use opentelemetry::propagation::TraceContext;
use opentelemetry_sdk::trace::TracerProvider;
use opentelemetry_stdout::SpanExporter;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;
/// Sets up a local tracing pipeline that prints spans to stdout.
fn setup_tracing() {
// Create a tracer provider that exports to the terminal.
let provider = TracerProvider::builder()
.with_simple_exporter(SpanExporter::default())
.build();
// Register the provider so other crates can find it.
let tracer = provider.tracer("minimal-otel-example");
// Tell OpenTelemetry to read/write standard W3C trace headers.
global::set_text_map_propagator(TraceContext::new());
// Wire the tracing subscriber to the OpenTelemetry layer.
tracing_subscriber::registry()
.with(tracing_opentelemetry::layer().with_tracer(tracer))
.init();
}
#[tokio::main]
async fn main() {
setup_tracing();
// Create a root span to represent the entire request lifecycle.
tracing::info!(parent: None, "starting application");
// Simulate work that would normally be an HTTP handler.
do_work().await;
}
/// Simulates a database call or external service request.
async fn do_work() {
// Create a child span to isolate this specific operation.
tracing::info!("processing payload");
tokio::time::sleep(tokio::time::Duration::from_millis(50)).await;
tracing::info!("payload processed");
}
Run this with RUST_LOG=info cargo run. You will see structured JSON output showing the trace ID, span ID, parent ID, and duration. The pipeline is live.
Walking through the data flow
When the program starts, setup_tracing() builds a TracerProvider. The provider holds the configuration for how spans are collected and exported. We attach a simple stdout exporter so every completed span prints immediately. The tracer object is the actual entry point for creating spans.
The global::set_text_map_propagator call registers a default context extractor and injector. This is crucial for cross-service communication. Without it, every service would generate isolated traces that never connect. The propagator looks for traceparent and tracestate headers in incoming requests, reconstructs the context, and pushes it onto the thread-local execution scope.
The tracing_subscriber::registry() call builds the actual logging infrastructure. We attach the tracing_opentelemetry::layer() to it. This layer sits between your tracing::info! macros and the final output. When a macro fires, the layer intercepts the event, checks if it should become a span, translates the metadata into OpenTelemetry format, and forwards it to the provider. The init() call installs this subscriber globally for the current thread.
Inside main, we call tracing::info!(parent: None, "starting application"). The parent: None hint tells the layer to create a root span. When do_work() runs, its tracing::info! calls automatically become child spans because they execute within the active execution context. The async runtime preserves this context across .await points, so the trace ID survives the sleep. When the function returns, the span closes, the duration is calculated, and the exporter flushes the data.
Wiring it into an Axum router
Production services need to capture HTTP metadata automatically. The axum-tracing-opentelemetry crate handles header extraction, span naming, and status code injection without manual boilerplate.
[dependencies]
axum = "0.7"
axum-tracing-opentelemetry = "0.10"
tracing = "0.1"
tracing-subscriber = "0.3"
tracing-opentelemetry = "0.23"
opentelemetry = "0.22"
opentelemetry-stdout = "0.3"
tokio = { version = "1", features = ["full"] }
use axum::{routing::get, Router};
use axum_tracing_opentelemetry::OpenTelemetryMiddleware;
use opentelemetry::global;
use opentelemetry::propagation::TraceContext;
use opentelemetry_sdk::trace::TracerProvider;
use opentelemetry_stdout::SpanExporter;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;
/// Configures the OpenTelemetry pipeline and returns the tracer.
fn init_otel() -> opentelemetry_sdk::trace::Tracer {
let provider = TracerProvider::builder()
.with_simple_exporter(SpanExporter::default())
.build();
let tracer = provider.tracer("axum-otel-service");
global::set_text_map_propagator(TraceContext::new());
tracing_subscriber::registry()
.with(tracing_opentelemetry::layer().with_tracer(tracer.clone()))
.init();
tracer
}
/// Handles the root endpoint and records a custom event.
async fn health_check() -> &'static str {
tracing::info!("health check requested");
"ok"
}
#[tokio::main]
async fn main() {
// Initialize the pipeline before any routes are built.
init_otel();
// Attach the middleware to automatically extract trace context.
let app = Router::new()
.route("/", get(health_check))
.layer(OpenTelemetryMiddleware::new());
// Bind to localhost and start serving.
let listener = tokio::net::TcpListener::bind("127.0.0.1:3000")
.await
.expect("failed to bind port");
axum::serve(listener, app).await.expect("server crashed");
}
The middleware does three things behind the scenes. It reads the traceparent header from the incoming request. It creates a root span named after the route pattern. It injects the span context into the request extensions so downstream handlers can access it. When the response finishes, the middleware records the HTTP status code and duration, then closes the span. You get a complete request lifecycle trace without writing a single line of tracing code in the handler.
Convention note: The OpenTelemetry Rust ecosystem moves quickly. Crate versions must align exactly. opentelemetry 0.22 pairs with tracing-opentelemetry 0.23 and opentelemetry-stdout 0.3. Mixing 0.21 and 0.22 will trigger trait bound errors because the internal Context and Tracer types changed. Always check the Cargo.toml of the middleware you are using and match its dependency versions.
Where things break and how to fix them
Version mismatches are the most common failure. The compiler will reject your code with E0277 (trait bound not satisfied) when the Tracer type from one crate does not implement the trait expected by another. Fix this by aligning all opentelemetry-* and tracing-* crates to the same minor release.
Forgetting to set the propagator breaks distributed tracing. Your local traces will look perfect, but cross-service calls will spawn new trace IDs instead of continuing the existing one. The fix is always global::set_text_map_propagator(TraceContext::new()) before the subscriber initializes.
Blocking the runtime with synchronous exporters kills throughput. The opentelemetry-stdout simple exporter flushes synchronously. In production, switch to opentelemetry-otlp with an async batch exporter. The batch exporter collects spans in memory and ships them in chunks, preventing network latency from stalling your request handlers.
Missing tracing feature flags silently disable macros. If you use tracing::info! but did not enable the std or log features in your Cargo.toml, the macros compile to no-ops. You will see zero output and no compiler warnings. Always enable tracing = { version = "0.1", features = ["std"] } unless you are building for no_std.
If you forget to call .init() on the subscriber, your spans vanish into the void. The compiler will not catch this. The runtime simply has no active subscriber to receive the events. Trust the initialization order. Set up the pipeline first. Build the router second. Start the server third.
Choosing your observability stack
Use OpenTelemetry when you need to track requests across multiple services, async tasks, or language boundaries. Use structured logging with tracing and tracing-subscriber when you only need local debugging and do not require cross-service correlation. Use Prometheus metrics with the metrics crate when you care about aggregate rates, histograms, and alerting thresholds rather than individual request paths. Reach for plain println! or log only in throwaway scripts where startup time and runtime overhead must be zero. Pick the tool that matches the scope of the problem. Observability is a cost. Pay for it only where the data will actually change your decisions.