Spans and Instrumentation: Request-Level Tracing Basics

A span is a named operation that represents a piece of work within a single service. Spans record timing, status, and contextual information (attributes). A trace is a collection of related spans across service boundaries. This article teaches you how to create spans in Rust, add attributes and events, and use automatic instrumentation via the tracing crate to emit spans with zero-copy overhead.

Spans are the foundation of distributed tracing. Unlike metrics (aggregated totals), spans capture the full detail of individual requests, enabling forensic debugging and latency analysis at the operation level.

Manual Span Creation

The low-level way to create a span is directly via the OpenTelemetry API:

use opentelemetry::global;
use opentelemetry::trace::{Span, Tracer};

let tracer = global::tracer("my_app");

// Create a span for a single operation
let mut span = tracer.start("process_request");

// Add attributes (metadata)
span.set_attribute(opentelemetry::KeyValue::new("user.id", "12345"));
span.set_attribute(opentelemetry::KeyValue::new("http.method", "POST"));

// Simulate work
std::thread::sleep(std::time::Duration::from_millis(50));

// Mark the span as complete
span.end();

Each span has:

• Name (e.g., "process_request", "database_query") — descriptive and reused across requests.
• Timing — automatically recorded from creation to .end().
• Attributes — key-value metadata (user ID, HTTP method, error type).
• Status — success or error.
• Events — optional discrete occurrences during the span (cache hit, retry attempt).

In async code, tracking manual spans is cumbersome because spans must outlive await points. The tracing crate and tracing-opentelemetry solve this elegantly.

Automatic Instrumentation with tracing

The tracing crate provides macros that emit events and create spans with zero boilerplate. The #[instrument] macro automatically creates a span for function execution:

use tracing::{info, instrument};
use opentelemetry::global;

#[instrument(name = "fetch_user", skip(db_conn))]
async fn fetch_user(user_id: u64, db_conn: &DatabaseConnection) -> Result<User> {
    info!("Fetching user from database");

    let user = db_conn.query("SELECT * FROM users WHERE id = ?", user_id).await?;

    info!(user_name = %user.name, "User retrieved successfully");

    Ok(user)
}

The #[instrument] macro:

• Creates a span named "fetch_user" when the function is called.
• Automatically adds function arguments as attributes (except those listed in skip()).
• Calls .end() on the span when the function returns.

skip(db_conn) excludes the db_conn argument from attributes (it is not Debug-printable or is too verbose).

Adding Attributes and Events

Within an instrumented function, you can add more detail:

use tracing::{info, warn, Span};
use opentelemetry::KeyValue;

#[instrument(name = "process_payment")]
async fn process_payment(order_id: u64, amount: f64) -> Result<()> {
    info!("Starting payment processing");

    // Get the current span and add custom attributes
    let span = tracing::Span::current();
    span.record("amount_cents", (amount * 100.0) as i64);

    // Conditional logging
    if amount > 1000.0 {
        warn!("Large payment detected");
        span.record("large_payment", true);
    }

    // Simulate payment API call
    match call_payment_api(order_id, amount).await {
        Ok(txn_id) => {
            span.record("transaction_id", txn_id);
            info!("Payment succeeded");
            Ok(())
        }
        Err(e) => {
            span.record("error", format!("{:?}", e));
            warn!("Payment failed: {}", e);
            Err(e)
        }
    }
}

async fn call_payment_api(order_id: u64, amount: f64) -> Result<String> {
    // Mock implementation
    Ok(format!("txn_{}", order_id))
}

The span.record() method adds attributes dynamically. The tracing::info! and tracing::warn! macros log events within the span.

Nested Spans and Child Operations

Functions called from an instrumented function automatically create child spans:

#[instrument(name = "place_order")]
async fn place_order(user_id: u64, items: Vec<String>) -> Result<OrderId> {
    info!("Placing order for user {}", user_id);

    // Call a function that also has #[instrument]
    // This creates a child span
    let address = fetch_user_address(user_id).await?;

    // Another child span
    let total = calculate_total(&items).await?;

    info!("Creating order record");
    let order_id = create_order_in_db(user_id, &items, &address, total).await?;

    Ok(order_id)
}

#[instrument(name = "fetch_user_address", skip(user_id))]
async fn fetch_user_address(user_id: u64) -> Result<String> {
    info!("Querying database for user address");
    // Simulated query
    tokio::time::sleep(tokio::time::Duration::from_millis(50)).await;
    Ok(format!("123 Main St, City, State"))
}

#[instrument(name = "calculate_total")]
async fn calculate_total(items: &[String]) -> Result<f64> {
    info!("Computing total price");
    let total: f64 = items.len() as f64 * 19.99;
    Ok(total)
}

When place_order is called, Jaeger (or another trace backend) will show:

place_order (100ms total)
├─ fetch_user_address (50ms)
├─ calculate_total (2ms)
└─ [remaining work]

The parent-child relationships are automatic: #[instrument] functions record their scope and automatically link to the parent span.

Setting Span Status and Recording Errors

By default, spans are marked as Ok. If an operation fails, explicitly record the error:

use tracing::{error, instrument};
use opentelemetry::trace::Status as OTelStatus;

#[instrument(name = "save_to_database")]
async fn save_to_database(record: &Record) -> Result<()> {
    match db.save(record).await {
        Ok(_) => {
            info!("Record saved");
            Ok(())
        }
        Err(e) => {
            error!("Database error: {:?}", e);
            
            // Explicitly mark the span as failed
            Span::current().set_status(OTelStatus::error(e.to_string()));
            
            Err(e)
        }
    }
}

In Jaeger's UI, failed spans are highlighted in red, making it easy to spot errors in your trace.

Attributes Best Practices

Use semantic conventions (https://opentelemetry.io/docs/specs/semconv/) for attribute names:

Attribute	Example	Use Case
http.method	GET, POST	HTTP requests
http.status_code	200, 500	HTTP responses
db.operation	SELECT, INSERT	Database queries
db.name	users	Database name
user.id	12345	User identification
error.type	ValidationError	Error classification
messaging.message_id	msg_789	Message/event ID

Following conventions makes your traces interoperable with industry dashboards and tools.

Instrumentation Pattern: Middleware

For HTTP servers, the cleanest pattern is middleware that instruments every request:

use axum::{
    Router, routing::get,
    middleware::Next,
    response::Response,
    http::Request,
};
use tracing::Instrument;
use std::time::Instant;

async fn tracing_middleware<B>(
    req: Request<B>,
    next: Next,
) -> Response {
    let method = req.method().to_string();
    let uri = req.uri().to_string();
    let start = Instant::now();

    // Span name is the method + path
    let span = tracing::info_span!("http", method = %method, uri = %uri);

    let response = next.run(req).instrument(span).await;

    let elapsed_ms = start.elapsed().as_secs_f64() * 1000.0;
    tracing::info!(elapsed_ms = elapsed_ms, status = ?response.status(), "request completed");

    response
}

#[tokio::main]
async fn main() {
    // Initialize tracing
    tracing_subscriber::fmt::init();

    let app = Router::new()
        .route("/api/users", get(|| async { "Users" }))
        .layer(axum::middleware::from_fn(tracing_middleware));

    let listener = tokio::net::TcpListener::bind("127.0.0.1:8080").await.unwrap();
    axum::serve(listener, app).await.unwrap();
}

Now every HTTP request creates a span with method, URI, and latency automatically.

Key Takeaways

• Spans represent individual operations; combine them into traces to understand end-to-end request flow.
• Use #[instrument] to automatically create and manage spans with zero boilerplate.
• Add attributes via span.record() for rich contextual data.
• Nest instrumented functions to automatically create parent-child span relationships.
• Use semantic conventions (http.method, db.operation) for compatibility.
• Mark errors explicitly with span.set_status() for error visibility.

Frequently Asked Questions

What is the overhead of creating a span?

Very low. The #[instrument] macro and tracing crate are designed for zero-cost when no subscriber is attached. With a subscriber (OpenTelemetry export), overhead is typically less than 5% CPU for moderate tracing rates.

Can I manually create spans without #[instrument]?

Yes, but it is more verbose. Use tracer.start() and .end() directly for one-off spans. For functions, #[instrument] is preferred because it is boilerplate-free and handles async correctly.

How many spans should a single request generate?

Typically 10-50 spans, depending on the request complexity. Each significant operation gets a span: HTTP handler, database query, cache lookup, external API call. More spans = more detail, but higher overhead.

Can I filter spans to reduce overhead?

Yes. Use the tracing-subscriber crate to set log levels and filters:

tracing_subscriber::fmt()
    .with_max_level(Level::INFO)  // only info and above
    .init();

Only spans at INFO level or above will be processed.

What if I want to add a span to existing code without #[instrument]?

Use the Span::in_scope() method:

let span = tracing::info_span!("my_operation");
let _guard = span.enter();
// Code here runs within the span

Or use the instrument() method on futures:

let future = some_async_operation().instrument(tracing::info_span!("operation"));

Further Reading

• tracing Crate Documentation
• tracing-opentelemetry Integration
• OpenTelemetry Semantic Conventions
• Span Best Practices