Trace-Correlated Logs

[!IMPORTANT] Logs without trace IDs are just noise. Logs with trace IDs are a precision instrument.

Imagine a production outage. You have thousands of error logs flooding your console. Payment failed, Database timeout, NullPointerException. But which error belongs to which user request? Are they related?

Without trace correlation, you are left grepping by timestamps and praying for a match. With Trace-Correlated Logs, every single log line carries the DNA of the transaction it belongs to: the trace_id and span_id.

1. The Power of Correlation

When you connect logs to traces, you unlock the ability to:

  1. Jump from Log to Trace: See the full journey of a request that generated an error.
  2. Filter by Trace: Isolate all logs for a specific transaction across all microservices.
  3. Contextualize Errors: Understand why a log event happened based on the span’s attributes.

Interactive: The Correlation Simulator

Toggle the switch below to see the difference between standard logging and trace-correlated logging.

Mode: Standard Logs (Disconnected)

Application Logs

Distributed Trace (Waterfall)

Gateway: /checkout
Auth: Verify Token
Payment: Charge
DB: Update Order

2. How It Works: The MDC Bridge

The magic happens via the Mapped Diagnostic Context (MDC) in Java or Context propagation in Go. The OpenTelemetry agent or SDK intercepts the current span context and injects it into the logging framework’s thread-local storage.

  1. Request Starts: OTel creates a Span with trace_id.
  2. Context Injection: OTel puts trace_id and span_id into MDC.
  3. Log Event: Logger reads from MDC and formats the string.
  4. Output: Log line contains the IDs.

3. Implementation Guide

1. Java (Logback + Spring Boot)

In Java, the OpenTelemetry Java Agent handles this automatically. You just need to configure your logging pattern.

Dependency (if using Spring Boot 3+): No extra dependency needed for the agent, but ensure you have logback-classic.

src/main/resources/logback-spring.xml:

<configuration>
  <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
      <!-- %X{trace_id} and %X{span_id} are populated by OTel Agent -->
      <pattern>
        %d{ISO8601} [%thread] %-5level %logger{36} - trace_id=%X{trace_id} span_id=%X{span_id} - %msg%n
      </pattern>
    </encoder>
  </appender>

  <root level="INFO">
    <appender-ref ref="CONSOLE"/>
  </root>
</configuration>

[!TIP] If you are not using the Java Agent, you must manually instrument using opentelemetry-logback-mdc-1.0.

2. Go (Slog)

In Go, context must be passed explicitly. We use a “Bridge” to connect OTel to slog.

Dependencies:

go get go.opentelemetry.io/contrib/bridges/otelslog
go get go.opentelemetry.io/otel

Implementation:

package main

import (
	"context"
	"os"

	"go.opentelemetry.io/contrib/bridges/otelslog"
	"go.opentelemetry.io/otel"
  "log/slog"
)

func main() {
  // 1. Create a logger that speaks OTel
  logger := otelslog.NewLogger("my-service")

  // 2. Set as default (optional)
  slog.SetDefault(logger)

  ctx := context.Background()
  tracer := otel.Tracer("my-tracer")

  // 3. Start a span
  ctx, span := tracer.Start(ctx, "process-order")
  defer span.End()

  // 4. Log with Context!
  // The logger extracts the trace_id from 'ctx'
  logger.InfoContext(ctx, "Processing payment",
    "amount", 99.00,
    "currency", "USD",
  )

  // Error logging with span recording
  // span.RecordError(err) // Don't forget this for traces!
}

[!NOTE] In Go, you must use InfoContext, ErrorContext, etc., and pass the ctx object. Standard logger.Info will NOT capture the trace ID because Go has no ThreadLocal equivalent.

4. Structured Logging (JSON)

Text logs are fine for humans, but machines prefer JSON. When sending logs to ELK (Elasticsearch), Loki, or Datadog, use JSON to avoid expensive parsing rules.

Java (Logstash Encoder)

Add dependency:

<dependency>
  <groupId>net.logstash.logback</groupId>
  <artifactId>logstash-logback-encoder</artifactId>
  <version>7.4</version>
</dependency>

Logback Config:

<appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
  <encoder class="net.logstash.logback.encoder.LogstashEncoder">
    <includeMdcKeyName>trace_id</includeMdcKeyName>
    <includeMdcKeyName>span_id</includeMdcKeyName>
  </encoder>
</appender>

Go (Slog JSON)

Go’s slog supports JSON natively.

handler := slog.NewJSONHandler(os.Stdout, nil)
logger := slog.New(handler)

// Middleware or helper needed to inject trace_id into standard JSON handler
// Or stick with otelslog which handles it automatically.

5. Best Practices

1. Context, Context, Context

Don’t just log “Error”. Log attributes.

// BAD
slog.Error("Database failed")

// GOOD
slog.ErrorContext(ctx, "Database query failed",
  "query.id", "q_123",
  "retry_count", 3,
  "duration_ms", 150,
)

2. Avoid High Cardinality in Messages

The log message should be static. The attributes should contain variable data.

  • ❌ log.info("User jules_123 logged in") → Creates millions of unique message patterns.
  • ✅ log.info("User logged in", "user_id", "jules_123") → One message pattern, easy to aggregate.

3. Sampling Considerations

If you sample traces (e.g., keep 10%), what happens to logs?

  • Head Sampling: If the trace is dropped, the trace_id still exists in the log, but looking it up in Jaeger will return 404.
  • Recommendation: Log everything (if affordable) or use tail-sampling in the Collector to keep logs for failed requests even if the trace was going to be dropped.

6. Summary

Trace-correlated logs are the bridge between the “what” (Logs) and the “where/when” (Traces). By ensuring every log line carries a trace_id, you transform your debugging process from a guessing game into a precision investigation.

Next, we will look at how to review what we’ve learned.