Distributed Tracing with Micrometer and OpenTelemetry

Introduction

In a single application a slow or failed request leaves its story in one log file and one stack trace. In a system of microservices one click may pass through a gateway and five services, each with its own log, each handling hundreds of other requests at the same moment. Which lines in which logs belong to that one click, and where did its three seconds go? Distributed tracing answers this by giving every incoming request an id that travels with it from service to service. Spring Boot does this with Micrometer Tracing, and with OpenTelemetry as the standard underneath. This tutorial adds tracing to two services, follows one request through both logs, looks at the header that carries the trace between them, prints the spans that would go to a tracing backend, and adds a span of its own. The code runs on Spring Boot 4.1.2 with Micrometer Tracing 1.7.2 and OpenTelemetry 1.62.0.

Traces and spans

Two terms carry the whole idea -
  • A trace is the complete journey of one request through the system. It has a trace id, created by the first service that sees the request and passed along to every other.
  • A span is one step of that journey: handling an HTTP request, making an HTTP call, running a query. A span has its own span id, the id of its parent span, a name, a start time and a duration.
The spans of one trace form a tree, and drawn against time the tree shows at a glance where the time went - One trace drawn as a timeline - a span for POST /orders in the order service covers the whole request; inside it a client span for the HTTP call, inside that the catalog service's span for GET /books, and afterwards a span for order pricing; all four share one trace id and each names its parent When a service calls another, it sends the trace id and its current span id along in an HTTP header. The called service continues the same trace and makes the sender's span the parent of its own. This is context propagation.

Micrometer and OpenTelemetry

Three names appear in this area, and they are layers, not rivals -
  • Micrometer Observation is the API that Spring's own code is instrumented with. Spring MVC, WebFlux, RestClient, JdbcClient and others report "an HTTP request was handled" as an observation, from which both a metric and a span are made.
  • Micrometer Tracing turns observations into spans and puts the ids into the logging context. It is a facade over a tracer library.
  • OpenTelemetry is the vendor-neutral standard: the tracer behind the facade, the format of the propagation header (W3C Trace Context), and OTLP, the protocol in which spans are sent to a backend such as Jaeger, Grafana Tempo or Zipkin, or to a commercial service.
Spring Boot 4 has one starter that brings the three together -
pom.xml (extract) - in both services
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-opentelemetry</artifactId>
</dependency>

Two services with tracing

The services are the order service and the catalog service of the earlier tutorials in this module. Their configuration differs only in name and port -
order-service: src/main/resources/application.properties
spring.application.name=order-service
server.port=8080
catalog.url=http://localhost:8081

# trace every request (the default is 0.1 - one request in ten)
management.tracing.sampling.probability=1.0
# for this demonstration: hand finished spans to the exporter every second (default 5s)
management.opentelemetry.tracing.export.schedule-delay=1s
logging.level.com.javabrahman=INFO
catalog-service: src/main/resources/application.properties
spring.application.name=catalog-service
server.port=8081

management.tracing.sampling.probability=1.0
management.opentelemetry.tracing.export.schedule-delay=1s
logging.level.com.javabrahman=INFO
The catalog service logs one line per request, including the tracing header it received -
catalog-service: src/main/java/.../catalog/CatalogApp.java
//CatalogApp.java
package com.javabrahman.cloud.tracing.catalog;

import com.javabrahman.cloud.tracing.shared.ConsoleSpanExporter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Import;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestHeader;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@Import(ConsoleSpanExporter.class)
@RestController
public class CatalogApp {

  private static final Logger log = LoggerFactory.getLogger(CatalogApp.class);

  record Book(String isbn, String title, int price) {}

  @GetMapping("/books/{isbn}")
  Book book(@PathVariable String isbn, @RequestHeader("traceparent") String traceparent) {
    log.info("Looking up book {} - traceparent header: {}", isbn, traceparent);
    return new Book(isbn, "Effective Java", 450);
  }

  public static void main(String[] args) {
    SpringApplication.run(CatalogApp.class, args);
  }
}
The order service calls it with a RestClientClick to read tutorial: Calling REST Services with RestClient and HTTP Interfaces, and there is one rule to follow: build the client from the RestClient.Builder that Spring Boot provides. That builder is instrumented; a client made with RestClient.create() sends no tracing header and the trace ends at this service -
order-service: src/main/java/.../order/OrderApp.java
//OrderApp.java
package com.javabrahman.cloud.tracing.order;

import com.javabrahman.cloud.tracing.shared.ConsoleSpanExporter;
import io.micrometer.observation.Observation;
import io.micrometer.observation.ObservationRegistry;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Import;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestClient;

@SpringBootApplication
@Import(ConsoleSpanExporter.class)
@RestController
public class OrderApp {

  private static final Logger log = LoggerFactory.getLogger(OrderApp.class);

  record OrderRequest(String isbn, int quantity) {}
  record Book(String isbn, String title, int price) {}
  record Order(String title, int quantity, int total) {}

  private final RestClient catalog;
  private final ObservationRegistry observations;

  OrderApp(RestClient.Builder builder, @Value("${catalog.url}") String catalogUrl, ObservationRegistry observations) {
    this.catalog = builder.baseUrl(catalogUrl).build();
    this.observations = observations;
  }

  @PostMapping("/orders")
  Order place(@RequestBody OrderRequest request) {
    log.info("Placing an order for {} x {}", request.quantity(), request.isbn());
    Book book = catalog.get().uri("/books/{isbn}", request.isbn()).retrieve().body(Book.class);

    // a span of our own around a piece of business logic
    int total = Observation.createNotStarted("order.pricing", observations)
        .lowCardinalityKeyValue("currency", "INR")
        .observe(() -> priceWithDiscount(book.price(), request.quantity()));

    log.info("Order total is {}", total);
    return new Order(book.title(), request.quantity(), total);
  }

  private int priceWithDiscount(int price, int quantity) {
    log.info("Pricing {} copies", quantity);
    int total = price * quantity;
    return quantity >= 3 ? total * 90 / 100 : total;
  }

  public static void main(String[] args) {
    SpringApplication.run(OrderApp.class, args);
  }
}
Nothing in either controller mentions a trace. The only tracing code is the Observation around the pricing, which is optional and explained below.

An exporter that prints the spans

Finished spans are handed to a span exporter, which normally sends them to a tracing backend. No backend is used in this tutorial. To see exactly what a backend would receive, both services register this small exporter, which prints each span on the console; Spring Boot picks up any SpanExporter bean -
ConsoleSpanExporter.java - the same class in both services
//ConsoleSpanExporter.java
package com.javabrahman.cloud.tracing.shared;

import io.opentelemetry.sdk.common.CompletableResultCode;
import io.opentelemetry.sdk.trace.data.SpanData;
import io.opentelemetry.sdk.trace.export.SpanExporter;
import java.util.Collection;
import org.springframework.stereotype.Component;

@Component
public class ConsoleSpanExporter implements SpanExporter {

  @Override
  public CompletableResultCode export(Collection<SpanData> spans) {
    for (SpanData span : spans) {
      String parent = span.getParentSpanContext().isValid() ? span.getParentSpanId() : "none";
      long micros = (span.getEndEpochNanos() - span.getStartEpochNanos()) / 1_000;
      System.out.printf("SPAN %-8s %-22s trace=%s span=%s parent=%s %d us%n",
          span.getKind(), span.getName(), span.getTraceId(), span.getSpanId(), parent, micros);
    }
    return CompletableResultCode.ofSuccess();
  }

  @Override
  public CompletableResultCode flush() {
    return CompletableResultCode.ofSuccess();
  }

  @Override
  public CompletableResultCode shutdown() {
    return CompletableResultCode.ofSuccess();
  }
}

One request, two logs

Both services are started and one order is placed -  OUTPUT of placing an order
> curl -i -X POST http://localhost:8080/orders -H "Content-Type: application/json" -d '{"isbn":"978-1","quantity":3}'
HTTP/1.1 200
Content-Length: 52
Content-Type: application/json

{"title":"Effective Java","quantity":3,"total":1215}
The response says nothing about tracing. The logs do. First the log lines of the order service, then those of the catalog service -  OUTPUT in the log of the order service
2026-11-10T10:53:45.181+05:30  INFO 44320 --- [order-service] [nio-8080-exec-2] [5b79b82308fd46f13a9b2b3fcf1cc8b2-feca877e844d6af8] c.j.cloud.tracing.order.OrderApp         : Placing an order for 3 x 978-1
2026-11-10T10:53:46.079+05:30  INFO 44320 --- [order-service] [nio-8080-exec-2] [5b79b82308fd46f13a9b2b3fcf1cc8b2-c5b45ca986283bbf] c.j.cloud.tracing.order.OrderApp         : Pricing 3 copies
2026-11-10T10:53:46.082+05:30  INFO 44320 --- [order-service] [nio-8080-exec-2] [5b79b82308fd46f13a9b2b3fcf1cc8b2-feca877e844d6af8] c.j.cloud.tracing.order.OrderApp         : Order total is 1215
 OUTPUT in the log of the catalog service
2026-11-10T10:53:45.776+05:30  INFO 27956 --- [catalog-service] [nio-8081-exec-2] [5b79b82308fd46f13a9b2b3fcf1cc8b2-2a6fc0b2220ede47] c.j.cloud.tracing.catalog.CatalogApp     : Looking up book 978-1 - traceparent header: 00-5b79b82308fd46f13a9b2b3fcf1cc8b2-faff3cbe9e24edfc-03
Explanation of the output
  • With tracing on the classpath, Spring Boot adds a bracket of the form [traceId-spanId] to every log line, after the application name and the thread.
  • The trace id, the 32 characters before the hyphen, is the same in all four lines, across two processes with different process ids. Searching the logs of all services for this one value finds everything that happened for this request.
  • In the order service the first and third lines carry the same span id: that of the span for handling POST /orders. The line "Pricing 3 copies" in between carries another, because it was logged inside the custom span.
  • The catalog service's line has the same trace id and a span id of its own.
  • The traceparent header that the catalog service received has four parts separated by hyphens: a version (00), the trace id, the id of the calling span, and flags, in which the lowest bit says that this trace is sampled. This is the W3C Trace Context format, understood by tracing libraries in every language, so the chain also holds when the next service is not written with Spring.
  • The span id in the header appears in no log line of the order service. It belongs to the span that RestClient created for the outgoing call, and nothing was logged while that call was in progress. It shows up in the next output.

The spans

A second or so after the request, the exporters printed the finished spans. Again the order service first -  OUTPUT of the span exporter in the order service
SPAN CLIENT   http get               trace=5b79b82308fd46f13a9b2b3fcf1cc8b2 span=faff3cbe9e24edfc parent=feca877e844d6af8 833334 us
SPAN INTERNAL order.pricing          trace=5b79b82308fd46f13a9b2b3fcf1cc8b2 span=c5b45ca986283bbf parent=feca877e844d6af8 5514 us
SPAN SERVER   http post /orders      trace=5b79b82308fd46f13a9b2b3fcf1cc8b2 span=feca877e844d6af8 parent=none 1324269 us
 OUTPUT of the span exporter in the catalog service
SPAN SERVER   http get /books/{isbn} trace=5b79b82308fd46f13a9b2b3fcf1cc8b2 span=2a6fc0b2220ede47 parent=faff3cbe9e24edfc 340611 us
Explanation of the output
  • Each line is one span: its kind, its name, the trace id, its own id, its parent's id and its duration in microseconds. A span is exported when it ends, so the span that started first is printed last.
  • The order service produced three spans. http post /orders of kind SERVER has no parent: it is the root of the trace. The CLIENT span http get for the outgoing call and the INTERNAL span order.pricing both name the root as their parent.
  • The catalog service produced one SERVER span, and its parent is the CLIENT span of the order service - the id that arrived in the traceparent header. This link across the two processes is what joins the four spans into one tree.
  • Span names use the route pattern, /books/{isbn}, and not the actual path, so that all requests to one endpoint are grouped together.
  • The durations nest: the root span covers the whole request, the client span most of it, and the catalog's server span is shorter than the client span around it. The difference between those two is time spent outside the catalog's code - connection setup and the network. A tracing backend draws exactly this as a timeline.
Context propagation - the order service handles POST /orders in a server span, its RestClient creates a client span and sends the header traceparent with the trace id and that span's id; the catalog service reads the header, continues the same trace and creates its server span with the client span as parent; both services export their spans to a tracing backend

Spans of your own

Spring creates spans for what it knows: HTTP requests in and out, and with the right starters also database calls and messages. A slow step inside your own code is invisible unless it gets a span. The order service wrapped its pricing in an Observation -
  • Observation.createNotStarted("order.pricing", registry) names the observation, and observe(...) runs the code inside it: started before, stopped after, and marked as failed if the code throws.
  • One observation gives both a span in the trace and a timer metric of the same name. That is the reason to use the Observation API and not the tracer directly.
  • lowCardinalityKeyValue adds a tag with few possible values, safe for metrics. highCardinalityKeyValue is for values such as an order id, which go to the span only.
  • The annotation @Observed on a method does the same declaratively. It needs the AspectJ starter and management.observations.annotations.enabled=true.
In the output above, the log line "Pricing 3 copies" carries a different span id from the lines before and after it - the id of the order.pricing span - and the exporter shows that span as a child of the request's server span.

Sending traces to a backend

In production the spans go to a backend that stores them and draws the timeline. With the starter in place this needs one property; the address below is the usual one of an OpenTelemetry Collector or Jaeger on the same machine. It was not run for this tutorial -
application.properties - exporting over OTLP (not run here)
management.opentelemetry.tracing.export.otlp.endpoint=http://localhost:4318/v1/traces
management.tracing.sampling.probability=0.1
Points to decide -
  • Sampling - tracing every request of a busy system costs storage and some time. The default records one request in ten. The decision is made once, by the first service, and travels in the header, so a trace is recorded either in all services or in none.
  • Every hop must take part - a service or gateway that does not forward the header cuts the trace in two. Spring Cloud GatewayClick to read tutorial: API Gateway with Spring Cloud Gateway forwards and joins traces when the same starter is added to it.
  • Threads and messages - the trace context lives in the current thread. It follows a RestClient or WebClient call by itself; work handed to another thread or sent through a message broker needs the instrumented executors and templates that Spring provides, or the trace stops there.
  • Logs - keep the trace id in every log line, as above. A log search for one trace id then shows the lines of all services for one request, even without a tracing backend.

Summary

In this tutorial we added spring-boot-starter-opentelemetry to two services and wrote no tracing code for the basics. One request produced log lines in both services with the same trace id; the traceparent header carried that id, and the calling span's id, from the order service to the catalog service. A small exporter printed the spans: a server span per service, a client span for the call between them, and a custom span from the Observation API, linked into one tree by their parent ids. Exporting to a real backend is one OTLP property away. Tracing shows how services talk to each other; the next tutorial compares the ways they can: REST, messaging and gRPCClick to read tutorial: Inter-Service Communication: REST vs Messaging vs gRPC.