Inside Prometheus Metrics: Deep Architecture, Types, and Storage

A Prometheus metric is an immutable time series stream of float64 samples paired with millisecond-resolution timestamps, uniquely identified by a metric name and an arbitrary set of key-value label dimensions. In high-throughput distributed systems, misunderstanding how the Prometheus TSDB ingests, stores, and evaluates these streams causes runaway memory consumption, misleading dashboards, and silently broken alerting pipelines.

When an application exposes metric endpoints under heavy load, naive instrumentation choices quickly reveal their real costs. Unbounded label values trigger cardinality explosions that destabilize the TSDB Head chunk allocator, while misconfigured histogram buckets yield interpolation errors exceeding 300 percent during latency spikes. Selecting between Counters, Gauges, Histograms, and modern Native Histograms is not merely an API preference; it dictates whether your operational monitoring stays responsive or collapses under its own overhead.

This technical guide dissects the internal mechanics of Prometheus metrics, ranging from the byte-level exposition format and TSDB compression to the mathematical reality of counter resets and native sparse histograms in Prometheus v3.

Core Architecture of Prometheus Metrics and the Exposition Format

The foundation of Prometheus operational telemetry relies on an inverted, pull-based collection model. Rather than forcing application threads to push metrics asynchronously over UDP or TCP to a centralized ingest gateway, the Prometheus server initiates outbound HTTP GET requests against defined targets at designated scrape intervals. This architectural choice decouples application performance from telemetry backend availability and inherently regulates ingestion backpressure at the scraper level.

Every scraped target exposes its state through an HTTP text exposition endpoint, typically mapped to /metrics. The Prometheus TSDB conceptualizes all incoming telemetry through a strictly defined identity model: a metric identity is the combined set of the metric name and its unique, sorted key-value pairs (known as labels). Any permutation in label key or value defines an entirely separate time series with its own independent chunk buffer in memory.

+-------------------------------------------------------------+
| Target Application |
| |
| +-------------------+ +--------------------------+ |
| | In-Memory Counter | | OpenMetrics Formatter | |
| | / Gauge State | ====> | (Generates text payload) | |
| +-------------------+ +--------------------------+ |
+---------------------------------------------|---------------+
 | HTTP GET /metrics
 v
+-------------------------------------------------------------+
| Prometheus TSDB Engine |
| |
| +-------------------+ +--------------------------+ |
| | Stream Parser | ====> | Head Block (RAM Chunks) | |
| | (Zero-copy parse) | | Gorilla XOR / Delta-v | |
| +-------------------+ +--------------------------+ |
| | |
| v |
| +--------------------------+ |
| | WAL (Write-Ahead Log) | |
+-------------------------------------------------------------+

Inside the TSDB engine, each sample written to a time series consumes an initial footprint in the Head block. Timestamps are encoded using delta-of-delta compression, which reduces 64-bit integer values to an average of just over one byte per sample. Numeric values are compressed via Gorilla XOR floating-point encoding, discarding leading and trailing zero bits to achieve between 1.3 and 1.7 bytes per raw metric point.

Anatomy of a Raw Exposition Payload

A production prometheus metrics example follows a rigid, line-delimited syntax standardized under the OpenMetrics specification. Every metric stream is declared with explicit metadata comments before the series samples appear:

# HELP http_requests_total Total number of HTTP requests processed.
# TYPE http_requests_total counter
http_requests_total{method="POST",handler="/api/v1/checkout",status="200"} 104523 1774886400000
http_requests_total{method="POST",handler="/api/v1/checkout",status="500"} 412 1774886400000

# HELP jvm_memory_used_bytes Current memory usage in bytes.
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{area="heap",id="G1 Eden Space"} 8589934592 1774886400000

Dissecting this payload demonstrates how Prometheus processes incoming data:

• # HELP metadata: A human-readable text string describing the metric. The Prometheus server strips this during scrape processing to save TSDB metadata memory unless explicitly preserved in query interfaces.
• # TYPE metadata: Informs the parser how to treat the incoming data structure. If the type is omitted, the scraper defaults the metric to an untyped series, disabling type-specific validation rules.
• Label Dimensions: Comma-separated key-value pairs wrapped in curly braces. Every unique key-value combination constitutes a new 64-bit series hash inside the Prometheus inverted index.
• Sample Value: A 64-bit floating-point numeric value (float64). Prometheus cannot store arbitrary strings or raw JSON objects as metric values; non-numeric metadata must be modeled as label values.
• Optional Timestamp: An explicit Unix epoch timestamp in milliseconds. If omitted, the Prometheus scraper assigns the precise timestamp of when the network scrape took place.

Architecture Rule: Never omit the # TYPE metadata annotation in custom expositions. While Prometheus can parse untyped metrics, downstream tools, PromQL linters, and OpenMetrics-compliant parsers rely on explicit typing to validate monotonic assumptions and calculate distribution quantiles accurately.

Deconstructing the Core Prometheus Metric Types

Telemetry engineers model real-world state using four fundamental prometheus metric types, complemented by extended classifications from the OpenMetrics specification. Although the underlying TSDB stores every metric as a series of float64 numbers regardless of classification, the assigned type dictates client-side validation, aggregation safety, and PromQL evaluation behaviors.

Taxonomy of Supported Metric Types

• Counter: A cumulative metric representing a single monotonically increasing value. It can only increase or be reset to zero upon application restart. Counters are mathematically suited for tracking event counts, processed payloads, and cumulative errors.
• Gauge: A snapshot metric that fluctuates arbitrarily up and down. Gauges reflect instantaneous values such as memory consumption, queue depths, active worker threads, and CPU temperature.
• Histogram: A client-configured statistical distribution that samples observations (usually request durations or response sizes) and counts them into cumulative bucket series. It exposes an individual counter for each defined boundary, alongside a sum of all observed values and a count of total events.
• Summary: A client-calculated statistical distribution that exposes observations alongside streaming phi-quantiles (such as P50, P90, P99) calculated directly in application memory over a sliding temporal window.
• Info (OpenMetrics): A pseudo-counter initialized to a value of 1, used entirely to expose static metadata (such as application version, Git commit hash, or operating system architecture) across label dimensions without polluting dynamic state metrics.
• StateSet (OpenMetrics): A multi-state representation that acts as a set of booleans over a predefined finite state machine, exposing a 0 or 1 value for each possible state (such as cluster_node_state{state="leader"} 1).

Metric Types Operational Matrix

Production Selection Criteria

• Use a Counter whenever the primary operational question requires measuring the rate of change, frequency of occurrences, or total throughput over a defined timeframe.
• Select a Gauge only when the current instantaneous measurement is meaningful without computing a delta against past observations.
• Deploy a Histogram when measuring distributions across a fleet of service instances where aggregation across nodes is required to evaluate global service level indicators (SLIs).
• Avoid classical Summaries if your architecture relies on multi-replica clustering, because client-calculated percentiles cannot be combined across distinct scrape targets without statistical invalidation.

Prometheus Gauge vs Counter: State Tracking vs Monotonic Cumulative Values

The distinction between a prometheus gauge and a counter is the most critical operational boundary in time-series modeling. Confusing these two types leads to silent calculation errors in alerts and catastrophic misinterpretations of production health. A counter models cumulative work completed; a gauge models the current saturation of a resource.

Counter Resets and PromQL Compensation Mechanics

The foundational characteristic of a counter is monotonic progression. Because counters only tick upward, any decrease in a counter’s raw reported value indicates an anomaly, almost universally an application process restart or container recreation. PromQL functions like rate(), irate(), and increase() explicitly watch for decreases between successive data points.

Sample Sequence: [100, 150, 180, 20, 50]
 ^
 Process restart detected
Calculated delta: (150-100) + (180-150) + (20-0 [reset assumed]) + (50-20) = 130

When rate() encounters a value smaller than the preceding sample, it assumes the counter reset to zero at some instant between the two scrapes. It adds the new sample value directly to the accumulated delta, preserving the integrity of velocity calculations over time windows that span across deployments.

The Danger of Naive Gauge Modeling

Engineers often make the mistake of using a prometheus gauge vs counter implementation to record event rates by computing deltas client-side or incrementing and decrementing an ongoing total. If an application attempts to monitor error frequency using a gauge that increments on failure and decrements after a background recovery loop, several failure modes occur:

• Scrape Blindness: Scrapes occur at discrete time intervals (for instance, every 15 seconds). If 50 errors occur and 50 recoveries happen between scrape ticks, the gauge reports 0. The incident is completely invisible to Prometheus.
• Restart Ambiguity: When a gauge drops from 200 to 0, Prometheus cannot distinguish whether the active thread pool drained naturally or if the process crashed and restarted.
• Extrapolation Hazards: PromQL functions such as deriv() and predict_linear() applied to erratic gauges introduce severe noise, whereas applying rate() to a gauge is mathematically invalid and produces nonsensical rate calculations.

Side-by-Side Code Comparison

Consider the correct and incorrect ways to track request volumes and memory consumption in Go:

package main

import (
 "github.com/prometheus/client_golang/prometheus"
 "github.com/prometheus/client_golang/prometheus/promauto"
)

// CORRECT: Counter for accumulating discrete request events
var httpRequestsTotal = promauto.NewCounterVec(
 prometheus.CounterOpts{
 Name: "http_requests_total",
 Help: "Total number of inbound HTTP requests.",
 },
 []string{"method", "status"},
)

// CORRECT: Gauge for dynamic resource saturation
var activeConnections = promauto.NewGauge(
 prometheus.GaugeOpts{
 Name: "tcp_active_connections",
 Help: "Current number of established TCP connections.",
 },
)

// ANTI-PATTERN: Never use a Gauge to count occurrences
var badRequestsCounter = promauto.NewGauge(
 prometheus.GaugeOpts{
 Name: "http_requests_flawed_gauge",
 Help: "DO NOT USE: Missing events between scrapes",
 },
)

Prometheus Histogram Mechanics: Buckets, Quantiles, and Native Sparse Formats

A classic prometheus histogram exposes a stream of discrete counters split across user-configured numerical ranges. Understanding histogram mechanics is essential because distributed latency monitoring cannot be solved by tracking averages. An average response time of 50ms can easily conceal a 99th percentile tail latency of 8,000ms that disrupts downstream systems.

Classic Bucketed Histograms Under the Hood

When an application samples an observation using a classic histogram, the client library updates three elements simultaneously:

1. A sequence of cumulative counter time series identified by the label le (less-than-or-equal-to).
2. A cumulative counter named <metric_name>_count tracking total observations.
3. A cumulative counter named <metric_name>_sum tracking the arithmetic sum of all observed values.

# Classic Histogram Exposition Payload
http_request_duration_seconds_bucket{le="0.05"} 24050
http_request_duration_seconds_bucket{le="0.1"} 32100
http_request_duration_seconds_bucket{le="0.25"} 38900
http_request_duration_seconds_bucket{le="0.5"} 40100
http_request_duration_seconds_bucket{le="1.0"} 40500
http_request_duration_seconds_bucket{le="+Inf"} 40512
http_request_duration_seconds_sum 3218.45
http_request_duration_seconds_count 40512

To calculate a target percentile (such as the 95th percentile) from these series, PromQL applies the histogram_quantile() function:

histogram_quantile(
 0.95,
 sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)

The Linear Interpolation Flaw

Prometheus does not preserve individual latency values inside the TSDB. When computing quantiles, histogram_quantile() assumes that samples within a given bucket are distributed uniformly along a straight line. If the bucket boundaries are spaced too far apart, this linear interpolation produces massive analytical errors.

Bucket Sizing Reality: If your P99 latency shifts from 105ms to 240ms, but your configured buckets jump from le="0.1" to le="0.5", Prometheus interpolates uniformly across that entire 400ms spread. The resulting dashboard value is a mathematical artifact of the bucket boundaries, not a reflection of your real user experience.

Native Sparse Histograms in Prometheus v3

To eliminate the trade-off between bucket accuracy and time-series explosion, Prometheus v3 standardizes Native Histograms (sparse histograms). Instead of declaring static bucket thresholds on client initialization, Native Histograms allocate dynamic exponential buckets within single TSDB chunks.

Classic Histogram: 10 buckets = 10 separate TSDB time series
Native Histogram: 160 high-resolution buckets = 1 single TSDB time series

Native Histograms use a power-of-two schema factor. Each power-of-two boundary is subdivided into 2^schema logarithmic buckets. If an application records latency observations ranging from microseconds to minutes, the client library only creates and transmits indices for buckets that actually contain data points. The zero-bucket handles small observations near zero without infinite precision penalties, cutting TSDB storage overhead by 70 to 90 percent while boosting quantile accuracy down to single-digit percentage margins.

Architectural Tradeoffs: Histogram vs Gauge and Summary for Latency Tracking

Telemetry designers must weigh critical trade-offs when selecting between a histogram vs gauge or summary for tracking performance distributions. Choosing the wrong mechanism can completely blind engineering teams to tail-latency degradation or overwhelm Prometheus memory through series cardinality.

Why Gauges Must Never Track Latency

Attempting to monitor service response times by continuously setting a gauge to the most recent request duration creates severe monitoring blind spots:

• Tail Obliteration: If a microservice executes 1,000 requests per second and one request blocks for 10 seconds while the remaining 999 resolve in 2ms, a gauge records whichever single request finished immediately before the scrape boundary. The chance of catching that 10-second failure on a 15-second scrape cycle is negligible.
• The Averaging Averages Fallacy: If you attempt to mitigate this by recording moving averages into a gauge client-side, aggregating those gauges across multiple nodes using PromQL’s avg() produces an unweighted average of averages, an operation that is statistically invalid and masks severe tail degradation in high-throughput nodes.

Histograms versus Summaries

When tracking distributions properly, the architectural dilemma shifts between Histograms and Summaries. Summaries calculate streaming quantiles locally using algorithms like CKMS directly in the target process memory. While this approach provides exact mathematical quantiles without bucket configuration overhead, it introduces a major operational limitation: Summaries cannot be aggregated.

Cluster Setup: 3 Replicas running behind a load balancer
Replica A: Reports P99 = 20ms (handling 10,000 req/sec)
Replica B: Reports P99 = 35ms (handling 10,000 req/sec)
Replica C: Reports P99 = 800ms (handling 50 req/sec)

Can you calculate global P99? 
NO: Mathematically impossible to combine streaming quantiles from A, B, and C.
With Histograms: Buckets from A, B, and C are summed directly before computing quantiles.

Latency Measurement Strategy Matrix

Architectural Selection Checklist

• Select Classic or Native Histograms if you run horizontal microservices and must track combined global SLAs, SLOs, or multi-tenant percentile latencies.
• Select Summaries only when working within single-process monolithic systems where multi-node aggregation is never required and local quantile precision must be mathematically exact without tuning bucket boundaries.
• Never deploy a Gauge for operational latency monitoring; restrict gauges entirely to resource levels like memory consumption, connection counts, or storage capacity.

Production Instrumentation Patterns and Cardinality Control

Writing telemetry that withstands high-throughput production environments requires strict adherence to metric naming conventions, unit standardization, and aggressive cardinality controls. Unchecked label dimensions can trigger exponential index growth within the TSDB, causing out-of-memory crashes and severe scrape timeouts.

Standardized Metric Naming Rules

Every metric name exposed to Prometheus must conform to core architectural best practices:

• Base Unit Suffixes: Always expose measurements in base scientific units (seconds, bytes, meters, ratios). Never use non-standard units like milliseconds, kilobytes, or hours. Append the unit name directly: http_request_duration_seconds, disk_read_bytes.
• Total Suffix for Counters: Every monotonic counter must end with the suffix _total (for example, process_cpu_seconds_total, api_errors_total) in accordance with the OpenMetrics standard.
• Clean Label Scoping: Labels should identify the dimensions of the measured event, not encode metadata into the metric name itself. Use http_requests_total{method="POST"} rather than http_post_requests_total.

Production Go Instrumentation

The following Go implementation demonstrates production-grade HTTP middleware instrumentation using the official client_golang library, incorporating thread-safe registry handling and explicit bucket configurations:

package telemetry

import (
 "net/http"
 "strconv"
 "time"

 "github.com/prometheus/client_golang/prometheus"
 "github.com/prometheus/client_golang/prometheus/promhttp"
)

type MetricsMiddleware struct {
 requestDuration *prometheus.HistogramVec
 inFlightRequests prometheus.Gauge
}

func NewMetricsMiddleware(reg prometheus.Registerer) *MetricsMiddleware {
 m:= &MetricsMiddleware{
 inFlightRequests: prometheus.NewGauge(
 prometheus.GaugeOpts{
 Name: "http_in_flight_requests",
 Help: "Current number of HTTP requests being served concurrently.",
 },
 ),
 requestDuration: prometheus.NewHistogramVec(
 prometheus.HistogramOpts{
 Name: "http_request_duration_seconds",
 Help: "Histogram of latencies for HTTP requests processed by the service.",
 Buckets: []float64{0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0},
 },
 []string{"handler", "method", "code"},
 ),
 }

 reg.MustRegister(m.inFlightRequests)
 reg.MustRegister(m.requestDuration)
 return m
}

func (m *MetricsMiddleware) WrapHandler(handlerName string, next http.Handler) http.Handler {
 return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 m.inFlightRequests.Inc()
 defer m.inFlightRequests.Dec()

 start:= time.Now()
 dww:= &delegatedResponseWriter{ResponseWriter: w, statusCode: http.StatusOK}
 
 next.ServeHTTP(dww, r)

 duration:= time.Since(start).Seconds()
 m.requestDuration.WithLabelValues(
 handlerName,
 r.Method,
 strconv.Itoa(dww.statusCode),
 ).Observe(duration)
 })
}

type delegatedResponseWriter struct {
 http.ResponseWriter
 statusCode int
}

func (d *delegatedResponseWriter) WriteHeader(code int) {
 d.statusCode = code
 d.ResponseWriter.WriteHeader(code)
}

Production Python Instrumentation

In Python services, the official prometheus_client provides WSGI and ASGI integrations. Use custom registries to avoid leaking default process metrics into specialized collector outputs:

import time
from prometheus_client import CollectorRegistry, Counter, Histogram, generate_latest

custom_registry = CollectorRegistry()

HTTP_REQUEST_DURATION = Histogram(
 'http_request_duration_seconds',
 'HTTP request latency in seconds.',
 ['route', 'method', 'status'],
 registry=custom_registry,
 buckets=(0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0)
)

EXCEPTIONS_TOTAL = Counter(
 'app_unhandled_exceptions_total',
 'Total unhandled runtime exceptions.',
 ['exception_type'],
 registry=custom_registry
)

def timing_middleware(route_name, method, func, *args, **kwargs):
 start_time = time.perf_counter()
 status_code = 200
 try:
 return func(*args, **kwargs)
 except Exception as exc:
 status_code = 500
 EXCEPTIONS_TOTAL.labels(exception_type=type(exc).__name__).inc()
 raise
 finally:
 elapsed = time.perf_counter() - start_time
 HTTP_REQUEST_DURATION.labels(
 route=route_name,
 method=method,
 status=status_code
 ).observe(elapsed)

Managing High-Cardinality Labels

Cardinality is the total number of unique time series stored in the TSDB inverted index. Calculating cardinality across label permutations is simple multiplication:

Total Series = [Metric Names] x [Methods (5)] x [Handlers (40)] x [Status Codes (10)] x [Users (100,000)]
Total Series = 1 x 5 x 40 x 10 x 100,000 = 200,000,000 series (TSDB OOM Collapse)

Cardinality Defense Checklist

• Never place unbounded or dynamic runtime values into metric labels. This includes user IDs, UUIDs, session hashes, email addresses, order IDs, or raw URL paths containing query strings.
• Normalize route paths inside instrumentation middleware before applying labels. Map /users/41289/profile to the parameterized template /users/:id/profile.
• Enforce scrape constraints in your Prometheus server configuration using sample_limit, label_limit, and label_value_length_limit directives per scrape job.
• Deploy metric_relabel_configs on the Prometheus scraper to drop runaway debug dimensions before series enter the Head chunk memory buffer.

Prometheus metrics deliver deep operational visibility into high-throughput systems, but their reliability depends entirely on sound instrumentation choices. By selecting Counters for monotonic event progression, reserving Gauges strictly for resource state snapshots, and leveraging Histograms to evaluate distribution percentiles without masking tail latencies, engineers can build resilient monitoring pipelines that remain stable under severe production load.

As you evolve your monitoring infrastructure in 2026, audit existing application endpoints for cardinality leaks, normalize your metric naming around base scientific units, and evaluate Prometheus v3 Native Histograms to gain precise latency tracking while dramatically lowering TSDB memory overhead.