Exporting Rust Metrics to Prometheus: Scraping and Queries

Prometheus is the industry standard for metrics storage and querying. OpenTelemetry's Prometheus exporter transforms your Rust application's metrics into Prometheus text format, which Prometheus scrapes at regular intervals. This article covers configuring the exporter, setting up Prometheus to scrape your application, and writing queries to extract insights.

The key insight is that Prometheus is pull-based: your application exposes a /metrics endpoint in Prometheus text format, and Prometheus polls it every 15 seconds (configurable). This is simpler and more reliable than push-based monitoring, where your app sends data to a central collector.

Understanding Prometheus Text Format

Prometheus metrics are exposed as plain text. A simple counter might look like:

# HELP http_requests_total Total HTTP requests handled
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1523
http_requests_total{method="POST",status="500"} 42

The format is:

• # HELP <name> <description> — optional metadata.
• # TYPE <name> <type> — declares the metric type (counter, gauge, histogram).
• <name>{label1=\"value1\",...} value — the actual metric.

When you initialize OpenTelemetry's Prometheus exporter, it handles this serialization for you. You only need to expose the endpoint.

Configuring the Prometheus Exporter in Rust

Using opentelemetry-prometheus, the minimal setup is:

use opentelemetry_prometheus::PrometheusBuilder;
use axum::{Router, routing::get};

#[tokio::main]
async fn main() {
    // Initialize Prometheus exporter
    let prometheus_exporter = PrometheusBuilder::new()
        .install_simple()
        .expect("Prometheus exporter failed");

    // Expose the /metrics endpoint
    let app = Router::new()
        .route("/metrics", get(move || async {
            prometheus_exporter.render()
        }));

    let listener = tokio::net::TcpListener::bind("127.0.0.1:9090")
        .await
        .expect("Failed to bind");

    axum::serve(listener, app).await.expect("Server error");
}

The .install_simple() method starts a background HTTP server on port :9090 by default. Alternatively, .build() returns a struct you can manually serve:

let (prometheus_exporter, _) = PrometheusBuilder::new()
    .build()
    .expect("Prometheus builder failed");

// Now manually expose /metrics on your own HTTP server
let app = Router::new()
    .route("/metrics", get(move || async {
        prometheus_exporter.render()
    }));

Call .render() to get the current snapshot of all metrics in Prometheus text format.

Custom Metric Prefixes and Namespacing

For large applications, prefix metrics to avoid collisions:

let prometheus_exporter = PrometheusBuilder::new()
    .with_registry_prefix("myapp")
    .with_default_summary_quantiles(vec![0.5, 0.95, 0.99])
    .install_simple()
    .expect("Prometheus exporter failed");

This prepends all metrics with myapp_, so requests_total becomes myapp_requests_total.

Setting Up Prometheus to Scrape Your Application

Prometheus itself is a separate service that scrapes your applications. Install Prometheus (via Docker, binary, or package manager) and configure it to scrape your Rust app.

Create a prometheus.yml:

global:
  scrape_interval: 15s  # Scrape every 15 seconds
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'my_rust_app'
    static_configs:
      - targets: ['localhost:9090']
    metrics_path: '/metrics'

Start Prometheus:

docker run -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

Or download the binary from https://prometheus.io/download/ and run:

./prometheus --config.file=prometheus.yml

Prometheus will scrape http://localhost:9090/metrics every 15 seconds and store the data locally. Visit http://localhost:9090 (Prometheus web UI) and click "Graph" to see your metrics.

Querying Metrics with PromQL

PromQL is Prometheus' query language. Here are essential patterns:

Simple selectors

# All requests
http_requests_total

# Requests with status=200
http_requests_total{status="200"}

# Requests with status=5xx (regex)
http_requests_total{status=~"5.."}

Range vectors and functions

# Rate of requests per second over the last 5 minutes
rate(http_requests_total[5m])

# Requests in the last minute
increase(http_requests_total[1m])

# Current value of a gauge
process_memory_usage_bytes

# 99th percentile latency
histogram_quantile(0.99, http_request_duration_ms_bucket)

Aggregation

# Sum of requests across all instances
sum(http_requests_total)

# Average latency
avg(http_request_duration_ms)

# Requests grouped by method
sum by (method) (http_requests_total)

Recording rules (optional, for computed metrics)

For frequently-used queries, define recording rules in Prometheus to precompute results:

groups:
  - name: custom_metrics
    interval: 30s
    rules:
      - record: http:requests:rate5m
        expr: rate(http_requests_total[5m])

      - record: http:latency:p99
        expr: histogram_quantile(0.99, http_request_duration_ms_bucket)

This creates two new metrics (http:requests:rate5m and http:latency:p99) that are automatically computed and stored.

Handling High-Cardinality Metrics

If you emit too many unique metric combinations (labels), Prometheus becomes slow and uses excessive memory. This happens when you use unbounded label values (user IDs, request paths, hostnames).

Example of bad cardinality:

// DON'T: user_id is unbounded (millions of users)
request_duration.record(elapsed_ms, &[
    KeyValue::new("user_id", user_id.to_string()),  // DANGER
    KeyValue::new("endpoint", "/api/items"),
]);

Better approach:

// DO: aggregate to a bounded dimension
request_duration.record(elapsed_ms, &[
    KeyValue::new("user_tier", user.tier),  // only "free", "pro", "enterprise"
    KeyValue::new("endpoint", "/api/items"),
]);

If you must track user-level metrics, use a time-series database like Grafana Loki or apply sampling.

Complete End-to-End Example

Here is a minimal HTTP service with Prometheus metrics:

use axum::{Router, routing::get, extract::Query};
use opentelemetry::global;
use opentelemetry_prometheus::PrometheusBuilder;
use serde::Deserialize;
use std::time::Instant;

#[derive(Deserialize)]
struct Params {
    delay_ms: Option<u64>,
}

#[tokio::main]
async fn main() {
    let prometheus = PrometheusBuilder::new()
        .install_simple()
        .expect("Prometheus failed");

    let meter = global::meter("http_server");
    let req_count = meter.u64_counter("http_requests_total").init();
    let req_duration = meter.f64_histogram("http_request_duration_ms").init();

    let app = Router::new()
        .route(
            "/api/process",
            get(move |Query(params): Query<Params>| async move {
                let start = Instant::now();
                
                // Simulate work
                if let Some(ms) = params.delay_ms {
                    tokio::time::sleep(
                        tokio::time::Duration::from_millis(ms)
                    ).await;
                }

                // Record metrics
                req_count.add(1, &[]);
                let elapsed = start.elapsed().as_secs_f64() * 1000.0;
                req_duration.record(elapsed, &[]);

                "Done"
            }),
        )
        .route("/metrics", get(move || async {
            prometheus.render()
        }));

    let listener = tokio::net::TcpListener::bind("127.0.0.1:9090").await.unwrap();
    axum::serve(listener, app).await.unwrap();
}

Run it:

cargo run
# In another terminal:
curl "http://localhost:9090/api/process?delay_ms=100"
curl http://localhost:9090/metrics

You will see metrics like:

http_requests_total 1
http_request_duration_ms_bucket{le="10"} 0
http_request_duration_ms_bucket{le="100"} 0
http_request_duration_ms_bucket{le="250"} 1
...
http_request_duration_ms_sum 105.2
http_request_duration_ms_count 1

Key Takeaways

• OpenTelemetry's Prometheus exporter converts Rust metrics to Prometheus text format automatically.
• Expose a /metrics endpoint that calls prometheus_exporter.render().
• Configure Prometheus to scrape your application endpoint at regular intervals (default 15s).
• Use PromQL to query metrics: rate(), increase(), histogram_quantile(), aggregations.
• Avoid high-cardinality labels (user IDs, arbitrary strings) to prevent Prometheus overload.

Frequently Asked Questions

What if my /metrics endpoint is slow?

The .render() call is usually fast (milliseconds), but if you have thousands of metrics, it might slow down. Cache the rendered output for a few seconds, or use a separate goroutine/thread pool. For high-volume services, consider using push-based exporters like OTLP instead.

Can I scrape multiple applications with one Prometheus?

Yes. Add multiple scrape_configs or use service discovery (Consul, Kubernetes) to dynamically add targets. Prometheus merges metrics from all targets.

What is the default scrape interval?

15 seconds (configurable in prometheus.yml). You can lower it to 10s for tighter feedback, but this increases Prometheus resource usage.

Can I export metrics to multiple backends simultaneously?

Yes. Create multiple exporter instances and call .render() on each. OpenTelemetry makes this straightforward. You can export to Prometheus and OTLP at the same time.

How long does Prometheus retain metrics?

By default, 15 days. You can configure retention in prometheus.yml:

global:
  retention_days: 30

For longer retention, use a remote backend like Thanos, Cortex, or TimescaleDB.

Further Reading

• Prometheus Official Documentation
• PromQL Query Language Guide
• Metric Naming and Labeling Best Practices
• OpenTelemetry Prometheus Exporter