Complete End-to-End Observability Example for Rust Apps

This article brings together everything from the series: metrics, traces, Prometheus export, Jaeger collection, and Grafana visualization. You will build a complete two-service example where Service A calls Service B, and you can observe the entire request flow with latency metrics, distributed traces, and dashboards.

The goal is a working blueprint you can adapt to your own microservices architecture.

Architecture Overview

You will create:

1. Service A (HTTP server on port 8080) — API gateway that handles user requests.
2. Service B (HTTP server on port 8081) — data service that processes items.
3. Prometheus (port 9090) — scrapes metrics from both services.
4. Jaeger (port 16686) — collects and visualizes traces.
5. Grafana (port 3000) — dashboards visualizing Prometheus metrics.

User Request
    ↓
Service A (metrics, traces)
    ↓ HTTP call with trace context
Service B (metrics, traces)
    ↓ (both export to Prometheus + Jaeger)
↙           ↘
Prometheus   Jaeger
    ↓           ↓
  Grafana    Jaeger UI

Project Setup

Create a Cargo workspace:

cargo new --bin complete-observability
cd complete-observability

In Cargo.toml:

[workspace]
members = ["service-a", "service-b", "common"]

[profile.release]
opt-level = 3

Create subprojects:

cargo new --bin service-a
cargo new --bin service-b
cargo new --lib common

In common/Cargo.toml:

[dependencies]
opentelemetry = { version = "0.23", features = ["rt-tokio"] }
opentelemetry-prometheus = "0.16"
opentelemetry-jaeger = { version = "0.22", features = ["rt-tokio"] }
tracing = "0.1"
tracing-opentelemetry = "0.25"
tracing-subscriber = { version = "0.3", features = ["fmt"] }
tokio = { version = "1", features = ["full"] }
axum = "0.7"
reqwest = { version = "0.11", features = ["json"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"

In service-a/Cargo.toml and service-b/Cargo.toml:

[dependencies]
common = { path = "../common" }
# ... (same as common/Cargo.toml)

Common Observability Setup Module

Create common/src/lib.rs:

use opentelemetry::global;
use opentelemetry_prometheus::PrometheusBuilder;
use opentelemetry_jaeger::new_pipeline;
use opentelemetry::sdk::Resource;
use opentelemetry::KeyValue;
use tracing_opentelemetry::OpenTelemetryLayer;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;

pub async fn init_observability(service_name: &str) -> PrometheusExporter {
    // Initialize Prometheus
    let prometheus_exporter = PrometheusBuilder::new()
        .install_simple()
        .expect("Prometheus exporter failed");

    // Initialize Jaeger with service name
    let resource = Resource::new(vec![
        KeyValue::new("service.name", service_name.to_string()),
        KeyValue::new("service.version", "1.0.0"),
    ]);

    let jaeger_tracer = new_pipeline()
        .install_simple()
        .expect("Jaeger exporter failed");

    // Setup tracing
    tracing_subscriber::registry()
        .with(tracing_subscriber::fmt::layer())
        .with(OpenTelemetryLayer::new(jaeger_tracer))
        .init();

    tracing::info!("Observability initialized for {}", service_name);
    prometheus_exporter
}

pub struct PrometheusExporter;

impl PrometheusExporter {
    pub fn render(&self) -> String {
        // Actual Prometheus rendering happens internally
        "".to_string()
    }
}

Service A: API Gateway

In service-a/src/main.rs:

use axum::{Router, routing::post, extract::Json, extract::State, middleware::Next,
           response::Response, http::Request};
use opentelemetry::global;
use serde::{Serialize, Deserialize};
use std::time::Instant;
use tracing::{info, instrument, Span};
use tower_http::trace::TraceLayer;

#[derive(Clone)]
struct AppState {
    client: reqwest::Client,
}

#[derive(Serialize, Deserialize)]
struct OrderRequest {
    user_id: u64,
    items: Vec<String>,
}

#[derive(Serialize, Deserialize)]
struct OrderResponse {
    order_id: String,
    total: f64,
}

async fn tracing_middleware<B>(
    req: Request<B>,
    next: Next,
) -> Response {
    let method = req.method().to_string();
    let uri = req.uri().to_string();
    let start = Instant::now();

    let span = tracing::info_span!("http_request", method = %method, uri = %uri);
    let _guard = span.enter();

    let response = next.run(req).await;

    let elapsed_ms = start.elapsed().as_secs_f64() * 1000.0;
    let meter = global::meter("service_a");
    let request_duration = meter.f64_histogram("http_request_duration_ms").init();
    let request_count = meter.u64_counter("http_requests_total").init();

    request_count.add(1, &[]);
    request_duration.record(elapsed_ms, &[]);

    info!(elapsed_ms = elapsed_ms, status = ?response.status(), "request completed");
    response
}

#[instrument(name = "place_order")]
async fn place_order(
    State(state): State<AppState>,
    Json(order): Json<OrderRequest>,
) -> Json<OrderResponse> {
    info!("Processing order for user {}", order.user_id);

    // Call Service B
    let items_json = serde_json::json!({"items": order.items});
    
    let response = state.client
        .post("http://localhost:8081/api/process-items")
        .json(&items_json)
        .send()
        .await
        .expect("Service B call failed");

    let total: f64 = response.json::<serde_json::Value>()
        .await
        .expect("Invalid response")["total"]
        .as_f64()
        .unwrap_or(0.0);

    let order_id = format!("ORD_{}", order.user_id);
    info!(order_id = %order_id, total = total, "order placed");

    Json(OrderResponse { order_id, total })
}

#[tokio::main]
async fn main() {
    common::init_observability("service-a").await;

    let state = AppState {
        client: reqwest::Client::new(),
    };

    let app = Router::new()
        .route("/api/order", post(place_order))
        .route("/metrics", axum::routing::get(|| async { "metrics" }))
        .with_state(state)
        .layer(axum::middleware::from_fn(tracing_middleware));

    let listener = tokio::net::TcpListener::bind("127.0.0.1:8080")
        .await
        .expect("Failed to bind");

    tracing::info!("Service A listening on 127.0.0.1:8080");
    axum::serve(listener, app).await.expect("Server error");
}

Service B: Data Service

In service-b/src/main.rs:

use axum::{Router, routing::post, extract::Json, middleware::Next,
           response::Response, http::Request};
use opentelemetry::global;
use serde::{Serialize, Deserialize};
use std::time::Instant;
use tracing::{info, instrument};

#[derive(Serialize, Deserialize)]
struct ItemsRequest {
    items: Vec<String>,
}

#[derive(Serialize, Deserialize)]
struct ItemsResponse {
    total: f64,
    count: usize,
}

async fn tracing_middleware<B>(
    req: Request<B>,
    next: Next,
) -> Response {
    let method = req.method().to_string();
    let uri = req.uri().to_string();
    let start = Instant::now();

    let span = tracing::info_span!("http_request", method = %method, uri = %uri);
    let _guard = span.enter();

    let response = next.run(req).await;

    let elapsed_ms = start.elapsed().as_secs_f64() * 1000.0;
    let meter = global::meter("service_b");
    let request_duration = meter.f64_histogram("http_request_duration_ms").init();
    let request_count = meter.u64_counter("http_requests_total").init();

    request_count.add(1, &[]);
    request_duration.record(elapsed_ms, &[]);

    info!(elapsed_ms = elapsed_ms, status = ?response.status(), "request completed");
    response
}

#[instrument(name = "process_items")]
async fn process_items(
    Json(req): Json<ItemsRequest>,
) -> Json<ItemsResponse> {
    info!("Processing {} items", req.items.len());

    // Simulate processing: calculate total price
    let total: f64 = req.items.iter().map(|_| 19.99).sum();

    info!(total = total, count = req.items.len(), "items processed");

    Json(ItemsResponse {
        total,
        count: req.items.len(),
    })
}

#[tokio::main]
async fn main() {
    common::init_observability("service-b").await;

    let app = Router::new()
        .route("/api/process-items", post(process_items))
        .route("/metrics", axum::routing::get(|| async { "metrics" }))
        .layer(axum::middleware::from_fn(tracing_middleware));

    let listener = tokio::net::TcpListener::bind("127.0.0.1:8081")
        .await
        .expect("Failed to bind");

    tracing::info!("Service B listening on 127.0.0.1:8081");
    axum::serve(listener, app).await.expect("Server error");
}

Docker Compose: Run Everything

Create docker-compose.yml:

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'

  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "6831:6831/udp"
      - "16686:16686"
      - "14268:14268"

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-storage:/var/lib/grafana

volumes:
  grafana-storage:

Create prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'service-a'
    static_configs:
      - targets: ['localhost:8080']

  - job_name: 'service-b'
    static_configs:
      - targets: ['localhost:8081']

Running the Complete Example

1. Start observability stack:

docker-compose up -d

2. Build and run services:

cargo build --release
./target/release/service-a &
./target/release/service-b &

3. Send a request:

curl -X POST http://localhost:8080/api/order \
  -H "Content-Type: application/json" \
  -d '{"user_id": 123, "items": ["item1", "item2"]}'

4. View metrics:

• Prometheus: http://localhost:9090 (query: http_requests_total)
• Grafana: http://localhost:3000 (add Prometheus data source)
• Jaeger: http://localhost:16686 (search for traces)

Observability Checklist

After running this example, verify:

• Both services export metrics to Prometheus.
• Prometheus scrapes /metrics endpoints successfully.
• Jaeger shows traces with both services linked.
• Trace context propagates from Service A to Service B (same trace ID).
• Latency histograms show p99 percentiles.
• Error handling (add a failing case, verify error spans in Jaeger).
• Grafana dashboard displays request rates and latencies.

Key Takeaways

• Use a shared observability module (common) for consistent initialization.
• Instrument HTTP middleware to automatically measure latency and count requests.
• Export metrics and traces from every service to the same backends.
• Use context propagation to link traces across service calls.
• Verify observability with Prometheus, Jaeger, and Grafana UIs.
• This structure scales to 10+ services with minimal changes.

Frequently Asked Questions

How do I add a third service?

Duplicate service-b, change the name and port, add it to docker-compose.yml and prometheus.yml. The tracing integration works automatically via W3C Trace Context headers.

Can I run this in Kubernetes?

Yes. Package each service as a Docker image, deploy as pods, and expose metrics via a Service. Use a DaemonSet to run Jaeger agents on each node, or deploy a single centralized collector.

What if I want to persist metrics?

Add a long-term-storage directory and configure Prometheus retention. For production, use Thanos, TimescaleDB, or a managed service (Grafana Cloud, Datadog).

How do I monitor the observability stack itself?

Prometheus, Jaeger, and Grafana expose their own metrics. Add them to your Prometheus scrape config to monitor their health (memory, requests, latency).

Can I add logging to this setup?

Yes. Use the tracing crate with tracing-subscriber to emit logs. Configure it to output JSON so you can ingest logs into Grafana Loki or Elasticsearch alongside metrics and traces.

Further Reading

• OpenTelemetry Rust Examples
• Complete Microservices Example
• Docker Compose for Observability Stack
• Production Observability Best Practices