Sampling Strategies at Scale

Part 8 of an 8-part series on implementing observability in Java microservices

Observability at scale has a dirty secret: It is expensive.

Logging every request, tracing every function, and counting every packet will bankrupt you. As traffic scales, your Observability bill often grows faster than your infrastructure bill. In this final module, we learn how to keep the signal but drop the noise.

1. The Cost Equation

The cost of tracing is directly proportional to the volume of data you ingest.

Cost ≈ Volume × (Storage + Compute + Network)

In a moderate system with 1,000 requests/sec, generating 10 spans per request:

  • 10,000 spans/sec
  • ~5KB per span
  • 50MB/sec → 4.3TB/day

You cannot store 4TB/day of trace data explicitly unless you have a massive budget. You must Sample.

Interactive: Sampling Cost Calculator

Adjust the sliders to see how quickly costs explode without sampling.

DATA VOLUME & COST ESTIMATOR
LIVE
10010k
150
0.102.00
Daily Volume
4.3 TB
Uncompressed
Monthly Bill
$38,880
Estimated

2. Sampling Strategies

1. Head Sampling (The “Coin Flip”)

Head sampling happens at the origin of the trace—usually in the SDK or Java Agent running inside your application. The decision is made before the first span is even exported.

How it Works

When a request hits your Frontend service, the SDK flips a coin (figuratively).

  • Heads: Sample (Record) the trace. The SDK attaches a sampled=true flag to the traceparent header.
  • Tails: Drop the trace. The SDK attaches sampled=false.

Crucially, this decision propagates downstream. If Frontend decides to drop a trace, Backend and Database will also drop it. This is called Parent-Based Sampling, and it ensures you don’t end up with “broken” traces where the child spans exist but the parent is missing.

Strategies

  1. always_on: Keep 100%. Great for development or low-traffic services.
  2. always_off: Keep 0%. Disables tracing.
  3. traceidratio: Probabilistic. Keeps a fixed percentage (e.g., 10%) based on the TraceID hash.
  4. parentbased_traceidratio (Recommended): Respects the parent’s decision if one exists. If it’s a new root trace, it uses the ratio.

Configuration (Java Agent)

# 1. Set the Sampler Strategy
# "parentbased_traceidratio" is the gold standard for production
OTEL_TRACES_SAMPLER=parentbased_traceidratio

# 2. Set the Ratio (Probability)
# 1.0 = 100% (Keep all), 0.1 = 10% (Keep 1 in 10), 0.001 = 0.1%
OTEL_TRACES_SAMPLER_ARG=0.1

# 3. Environment Tagging (Critical for filtering)
OTEL_RESOURCE_ATTRIBUTES=service.name=payment-service,deployment.environment=production

[!NOTE] Head sampling is efficient because dropped traces incur zero network or storage cost. However, it is “dumb”—it doesn’t know if a request will eventually fail. You might drop the most critical error of the day simply because the coin flip came up “tails”.

2. Tail Sampling (The “Smart Way”)

Tail sampling moves the decision making to the end of the workflow—specifically, to the OpenTelemetry Collector.

Because the Collector receives all spans, it can wait for the trace to complete (or time out) before deciding whether to keep it. This allows for intelligent policies:

  • Errors: “Keep any trace containing a span with status=ERROR.”
  • Latency: “Keep any trace that took > 2 seconds.”
  • Probabilistic: “Keep 1% of everything else.”

The Result: You keep 100% of the “interesting” data and drop 99% of the boring “success” data.

Interactive: The Sampling Duel

Watch how Head Sampling (random 10%) compares to Tail Sampling (smart policies) when handling a mix of success (green) and error (red) traffic.

STRATEGY COMPARISON: HEAD VS TAIL
HEAD SAMPLER (NAIVE)
0/0 ERRORS CAUGHT
0% DROP RATE
TAIL SAMPLER (SMART)
0/0 ERRORS CAUGHT
0% DROP RATE
HEAD (10%)
COIN FLIP
STORAGE
TAIL (Errors + 10%)
POLICY CHECK
STORAGE

3. Custom Samplers (Programmatic)

Sometimes, configuration isn’t enough. You might need a sampler that checks a feature flag, a customer tier (Gold/Silver), or a specific HTTP header. In these cases, you can write a Custom Sampler.

Java Implementation

In Java, you implement the Sampler interface.

package com.example.observability;

import io.opentelemetry.api.common.AttributeKey;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.context.Context;
import io.opentelemetry.sdk.trace.data.LinkData;
import io.opentelemetry.sdk.trace.samplers.Sampler;
import io.opentelemetry.sdk.trace.samplers.SamplingResult;
import io.opentelemetry.api.trace.SpanKind;
import java.util.List;

public class PremiumUserSampler implements Sampler {
  @Override
  public SamplingResult shouldSample(
      Context parentContext,
      String traceId,
      String name,
      SpanKind spanKind,
      Attributes attributes,
      List<LinkData> parentLinks) {

    // Check for a custom attribute or header
    String userTier = attributes.get(AttributeKey.stringKey("user.tier"));

    if ("gold".equals(userTier)) {
      // Always sample Gold users
      return SamplingResult.recordAndSample();
    }

    // Drop everyone else (or delegate to another sampler)
    return SamplingResult.drop();
  }

  @Override
  public String getDescription() {
    return "PremiumUserSampler";
  }
}

Go Implementation

In Go, you implement the sdktrace.Sampler interface.

package main

import (
	"fmt"
	"go.opentelemetry.io/otel/attribute"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/trace"
)

type PremiumUserSampler struct{}

func (s *PremiumUserSampler) ShouldSample(p sdktrace.SamplingParameters) sdktrace.SamplingResult {
	// iterate through attributes to find user.tier
	for _, attr := range p.Attributes {
		if attr.Key == attribute.Key("user.tier") && attr.Value.AsString() == "gold" {
			return sdktrace.SamplingResult{
				Decision:   sdktrace.RecordAndSample,
				Attributes: p.Attributes,
				Tracestate: trace.SpanContextFromContext(p.ParentContext).TraceState(),
			}
		}
	}

	// Drop otherwise
	return sdktrace.SamplingResult{
		Decision:   sdktrace.Drop,
		Attributes: p.Attributes,
		Tracestate: trace.SpanContextFromContext(p.ParentContext).TraceState(),
	}
}

func (s *PremiumUserSampler) Description() string {
	return "PremiumUserSampler"
}

4. The Architecture Challenge

Tail sampling implies state. To make a decision for TraceID: 0xABC, the Collector needs to see all spans for that trace.

If you have 5 Replica Collectors behind a standard Kubernetes Service (Round-Robin), spans for 0xABC will be scattered across different pods. No single collector has the full picture.

The 2-Tier Solution

You need a 2-Tier architecture to ensure Trace Affinity.

  1. Tier 1 (Gateway): Stateless. Receives spans from agents. Uses the loadbalancing exporter to Consistent Hash the TraceID and route it to a specific Tier 2 pod.
  2. Tier 2 (Sampler): Stateful. Receives all spans for 0xABC. Holds them in memory. Makes the decision.
PRODUCERS
App A
App B
gRPC (Round Robin)
TIER 1: GATEWAY (STATELESS)
Collector 1
Collector 2
Consistent Hash (TraceID)
TIER 2: SAMPLER (STATEFUL)
Sampler 1
Sampler 2 (Holds 0xABC)
Sampler 3

Configuration

Tier 1: The Gateway (Load Balancer)

# otel-collector-gateway.yaml
exporters:
  loadbalancing:
    protocol:
      port: 4317
    resolver:
      k8s: { service: "otel-collector-sampler" } # Points to Tier 2 Service
    routing_key: "traceID" # Ensure spans with same TraceID go to same destination

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [loadbalancing]

Tier 2: The Sampler (Decision Maker)

# otel-collector-sampler.yaml
processors:
  tail_sampling:
    decision_wait: 10s   # Wait 10s for spans to arrive
    num_traces: 50000    # Max traces in memory
    expected_new_traces_per_sec: 1000
    policies:
      - name: keep_errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: keep_slow_requests
        type: latency
        latency: {threshold_ms: 2000}
      - name: keep_1_percent
        type: probabilistic
        probabilistic: {sampling_percentage: 1}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp/honeycomb, logging] # Send to final backend

5. Remote Sampling (Jaeger Style)

What if you want the efficiency of Head Sampling but the control of centralized configuration?

Remote Sampling allows your application (SDK) to poll the Collector for sampling strategies. This means you can change sampling rates dynamically without redeploying your application.

  • SDK: “Hey Collector, what’s the sampling rate for payment-service?”
  • Collector: “It’s 5% right now.”

This is natively supported in the OpenTelemetry Jaeger Remote Sampler extension.

Configuration (Collector)

extensions:
  jaegerremotesampling:
    source:
      remote:
        endpoint: "jaeger-collector:14250"
    strategies:
      default_strategy:
        type: probabilistic
        param: 0.5
      service_strategies:
        - service: payment-service
          type: probabilistic
          param: 0.1 # 10% for payments
        - service: fraud-detection
          type: always_sample # 100% for fraud

service:
  extensions: [jaegerremotesampling]

```

3. Production Checklist

Before you declare victory, verify these 5 items:

1. Unified Service Naming

Ensure service.name is consistent across Metrics, Logs, and Traces. If your metrics say order-svc but your traces say OrderService, you cannot correlate them.

  • OTEL_SERVICE_NAME=order-service

2. Environment Tags

You will regret not knowing if a trace is from prod or staging.

  • OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.version=1.2.0

3. Queue Sizes

Monitor your Collector Queue. If it is always full, you are dropping data.

  • Check the metric otelcol_processor_batch_queue_length.
  • Increase send_batch_size or add more Collector replicas.

4. Health Checks

Do not trace your Kubernetes Health Checks (/health, /readiness). They are spam that consumes your sampling budget.

  • Java Agent: OTEL_JAVAAGENT_EXCLUDE_CLASSES=org.springframework.boot.actuate.health.HealthEndpoint
  • Collector: Use the filter processor to drop spans where http.target == "/health".

5. Secure your Exporters

Never expose 0.0.0.0:4317 (gRPC) to the public internet.

  • Use internal ClusterIPs in Kubernetes.
  • Enable mTLS if communicating across clusters.

4. Conclusion

We have built a world-class Observability stack. We started with Java Agents, moved to Manual Spans, added Baggage, integrated Metrics & Logs, built a Collector Pipeline, and optimized it with Tail Sampling.

You are now ready to debug anything.

Next Steps