Saga Pattern for Distributed Transactions

Introduction

Placing an order in a monolith is one database transaction: insert the order, take the payment, reduce the stock, and if anything fails, @TransactionalClick to read tutorial: Spring @Transactional: Propagation and Isolation rolls all of it back. Split the application into an order service, a payment service and an inventory service, each with its own database, and that safety is gone. No transaction spans three databases owned by three services. If the payment succeeds and the stock then turns out to be short, nothing rolls the payment back. The saga pattern is the standard answer: a sequence of local transactions, each with a compensating action that undoes it. This tutorial explains the pattern, builds a saga in its two forms - orchestration and choreography - and runs both through a success and two failures. The code runs on Spring Boot 4.1.2.

Why not one distributed transaction?

There is a protocol for transactions across several databases: two-phase commit (2PC, or XA). Microservices avoid it for good reasons -
  • Every participant holds its locks until all have voted and the coordinator has decided. One slow service slows down all, and a coordinator that fails at the wrong moment leaves the others locked and waiting.
  • All participants must be available at the same time, which multiplies their chances of failure.
  • Many things a service talks to - a REST API, a message broker, a payment provider - do not support it at all.
  • It requires services to reach into each other's transactions, which is exactly the coupling that microservices were meant to remove.

The saga

A saga gives up the single transaction and keeps the outcome. The business operation becomes a series of steps. Each step is an ordinary local transaction in one service and is committed at once. If a later step fails, the steps already done are not rolled back - they cannot be, they are committed - but compensated: for each of them a new transaction is run which reverses its effect, in the opposite order. A saga for placing an order - step 1 creates the order as pending, step 2 charges the payment, step 3 reserving the stock fails; the saga then runs the compensations backwards, first refunding the payment and then cancelling the order; every box is a separate local transaction in a different service
  • A compensation is a business action, not a database rollback. A charged payment is not erased; it is refunded, and both entries stay in the books.
  • Between the steps the system is in an in-between state that others can see: an order that is PENDING, a payment taken for stock not yet reserved. A saga has no isolation, and the design must allow for it. A status such as PENDING is the usual way.
  • The result is eventual consistency: at the end, either all steps are done, or all done steps are compensated.

The three services

The example has an order service, a payment service and an inventory service. To keep the tutorial runnable as one small program, the three are Spring beans in one application, with one in-memory H2 database. They are written as separate services would behave: each touches only its own tables, each method is a local transaction that commits by itself, and nothing spans two of them. In a real system each call from the saga to a service would be a REST call or a message, as described in the previous tutorialClick to read tutorial: Inter-Service Communication: REST vs Messaging vs gRPC.
src/main/resources/schema.sql
-- order service
create table orders (id bigint primary key, isbn varchar(20), quantity int, amount int, status varchar(20));
-- payment service
create table payments (order_id bigint primary key, amount int, status varchar(20));
-- inventory service
create table stock (isbn varchar(20) primary key, available int);
create table reservations (order_id bigint primary key, isbn varchar(20), quantity int);
insert into stock values ('978-1', 10), ('978-2', 3);
-- the orchestrator's own record of every saga
create table saga_log (id bigint auto_increment primary key, order_id bigint, step varchar(30), outcome varchar(120));
src/main/resources/application.properties
spring.main.banner-mode=off
logging.level.root=WARN
services/OrderService.java
//OrderService.java
package com.javabrahman.cloud.saga.services;

import java.util.concurrent.atomic.AtomicLong;
import org.springframework.jdbc.core.simple.JdbcClient;
import org.springframework.stereotype.Service;

@Service
public class OrderService {

  private final JdbcClient jdbc;
  private final AtomicLong ids = new AtomicLong(100);

  OrderService(JdbcClient jdbc) {
    this.jdbc = jdbc;
  }

  public long create(String isbn, int quantity, int amount) {
    long id = ids.incrementAndGet();
    jdbc.sql("insert into orders values (?, ?, ?, ?, 'PENDING')").params(id, isbn, quantity, amount).update();
    return id;
  }

  public void confirm(long orderId) {
    jdbc.sql("update orders set status = 'CONFIRMED' where id = ?").param(orderId).update();
  }

  // compensation of create()
  public void cancel(long orderId) {
    jdbc.sql("update orders set status = 'CANCELLED' where id = ?").param(orderId).update();
  }
}
services/PaymentService.java
//PaymentService.java
package com.javabrahman.cloud.saga.services;

import org.springframework.jdbc.core.simple.JdbcClient;
import org.springframework.stereotype.Service;

@Service
public class PaymentService {

  private static final int CARD_LIMIT = 2000;
  private final JdbcClient jdbc;

  PaymentService(JdbcClient jdbc) {
    this.jdbc = jdbc;
  }

  public void charge(long orderId, int amount) {
    if (amount > CARD_LIMIT) {
      throw new SagaStepException("payment declined: " + amount + " is over the card limit of " + CARD_LIMIT);
    }
    jdbc.sql("insert into payments values (?, ?, 'CHARGED')").params(orderId, amount).update();
  }

  // compensation of charge()
  public void refund(long orderId) {
    jdbc.sql("update payments set status = 'REFUNDED' where order_id = ?").param(orderId).update();
  }
}
services/InventoryService.java
//InventoryService.java
package com.javabrahman.cloud.saga.services;

import org.springframework.jdbc.core.simple.JdbcClient;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

@Service
public class InventoryService {

  private final JdbcClient jdbc;

  InventoryService(JdbcClient jdbc) {
    this.jdbc = jdbc;
  }

  @Transactional   // a local transaction: two statements in the inventory service's own tables
  public void reserve(long orderId, String isbn, int quantity) {
    int updated = jdbc.sql("update stock set available = available - ? where isbn = ? and available >= ?")
        .params(quantity, isbn, quantity).update();
    if (updated == 0) {
      throw new SagaStepException("out of stock: " + quantity + " copies of " + isbn + " are not available");
    }
    jdbc.sql("insert into reservations values (?, ?, ?)").params(orderId, isbn, quantity).update();
  }

  // compensation of reserve()
  @Transactional
  public void release(long orderId) {
    jdbc.sql("update stock set available = available + (select quantity from reservations where order_id = ?) "
        + "where isbn = (select isbn from reservations where order_id = ?)").params(orderId, orderId).update();
    jdbc.sql("delete from reservations where order_id = ?").param(orderId).update();
  }
}
services/SagaStepException.java
//SagaStepException.java
package com.javabrahman.cloud.saga.services;

public class SagaStepException extends RuntimeException {

  private static final long serialVersionUID = 1L;

  public SagaStepException(String message) {
    super(message);
  }
}
A small helper reads the state of all three services for one order, to print it after each run -
services/ShopState.java
//ShopState.java
package com.javabrahman.cloud.saga.services;

import org.springframework.jdbc.core.simple.JdbcClient;
import org.springframework.stereotype.Component;

@Component
public class ShopState {

  private final JdbcClient jdbc;

  ShopState(JdbcClient jdbc) {
    this.jdbc = jdbc;
  }

  public String of(long orderId, String isbn) {
    String order = jdbc.sql("select status from orders where id = ?").param(orderId).query(String.class).single();
    String payment = jdbc.sql("select status from payments where order_id = ?").param(orderId)
        .query(String.class).optional().orElse("none");
    int stock = jdbc.sql("select available from stock where isbn = ?").param(isbn).query(Integer.class).single();
    return "order " + orderId + " is " + order + " | payment: " + payment + " | stock of " + isbn + ": " + stock;
  }
}

Orchestration: one conductor

A saga needs something to decide what happens next. In orchestration that is one component, the orchestrator, which knows the whole sequence, calls each service in turn and, on a failure, calls the compensations. Saga orchestration - an orchestrator in the middle holds the list of steps and a saga log; it sends commands to the order service, the payment service and the inventory service one after another and receives each result; the services do not know each other The orchestrator below describes each step as a pair - an action and its compensation - and writes every outcome to the saga_log table -
orchestration/OrderSaga.java
//OrderSaga.java
package com.javabrahman.cloud.saga.orchestration;

import com.javabrahman.cloud.saga.services.InventoryService;
import com.javabrahman.cloud.saga.services.OrderService;
import com.javabrahman.cloud.saga.services.PaymentService;
import com.javabrahman.cloud.saga.services.SagaStepException;
import java.util.ArrayDeque;
import java.util.Deque;
import java.util.List;
import org.springframework.jdbc.core.simple.JdbcClient;
import org.springframework.stereotype.Component;

@Component
public class OrderSaga {

  record Step(String name, Runnable action, String undoName, Runnable compensation) {}

  private final OrderService orders;
  private final PaymentService payments;
  private final InventoryService inventory;
  private final JdbcClient jdbc;

  OrderSaga(OrderService orders, PaymentService payments, InventoryService inventory, JdbcClient jdbc) {
    this.orders = orders;
    this.payments = payments;
    this.inventory = inventory;
    this.jdbc = jdbc;
  }

  public long placeOrder(String isbn, int quantity, int unitPrice) {
    int amount = quantity * unitPrice;
    long orderId = orders.create(isbn, quantity, amount);
    log(orderId, "create-order", "DONE");

    List<Step> steps = List.of(
        new Step("charge-payment", () -> payments.charge(orderId, amount),
                 "refund-payment", () -> payments.refund(orderId)),
        new Step("reserve-stock", () -> inventory.reserve(orderId, isbn, quantity),
                 "release-stock", () -> inventory.release(orderId)));

    Deque<Step> completed = new ArrayDeque<>();
    for (Step step : steps) {
      try {
        step.action().run();
        log(orderId, step.name(), "DONE");
        completed.push(step);
      } catch (SagaStepException e) {
        log(orderId, step.name(), "FAILED - " + e.getMessage());
        compensate(orderId, completed);
        return orderId;
      }
    }
    orders.confirm(orderId);
    log(orderId, "confirm-order", "DONE");
    return orderId;
  }

  // undo the completed steps, the latest first, and finally the order itself
  private void compensate(long orderId, Deque<Step> completed) {
    while (!completed.isEmpty()) {
      Step step = completed.pop();
      step.compensation().run();
      log(orderId, step.undoName(), "COMPENSATED");
    }
    orders.cancel(orderId);
    log(orderId, "cancel-order", "COMPENSATED");
  }

  private void log(long orderId, String step, String outcome) {
    jdbc.sql("insert into saga_log (order_id, step, outcome) values (?, ?, ?)").params(orderId, step, outcome).update();
  }

  public List<String> history(long orderId) {
    return jdbc.sql("select step || ': ' || outcome from saga_log where order_id = ? order by id")
        .param(orderId).query(String.class).list();
  }
}
The application runs three orders. The first should succeed. The second is for five copies of a book of which three are in stock. The third costs more than the card allows -
orchestration/OrchestrationApp.java
//OrchestrationApp.java
package com.javabrahman.cloud.saga.orchestration;

import com.javabrahman.cloud.saga.services.ShopState;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;

@SpringBootApplication(scanBasePackages = {"com.javabrahman.cloud.saga.services", "com.javabrahman.cloud.saga.orchestration"})
public class OrchestrationApp {

  @Bean
  CommandLineRunner demo(OrderSaga saga, ShopState state) {
    return args -> {
      run(saga, state, "1. Two copies of 978-1 at 450", "978-1", 2, 450);
      run(saga, state, "2. Five copies of 978-2 at 100 (three in stock)", "978-2", 5, 100);
      run(saga, state, "3. Six copies of 978-1 at 450 (card limit 2000)", "978-1", 6, 450);
    };
  }

  private void run(OrderSaga saga, ShopState state, String title, String isbn, int quantity, int unitPrice) {
    System.out.println(title);
    long orderId = saga.placeOrder(isbn, quantity, unitPrice);
    saga.history(orderId).forEach(line -> System.out.println("   " + line));
    System.out.println("   => " + state.of(orderId, isbn));
    System.out.println();
  }

  public static void main(String[] args) {
    SpringApplication.run(OrchestrationApp.class, args);
  }
}
 OUTPUT of the orchestrated saga
1. Two copies of 978-1 at 450
   create-order: DONE
   charge-payment: DONE
   reserve-stock: DONE
   confirm-order: DONE
   => order 101 is CONFIRMED | payment: CHARGED | stock of 978-1: 8

2. Five copies of 978-2 at 100 (three in stock)
   create-order: DONE
   charge-payment: DONE
   reserve-stock: FAILED - out of stock: 5 copies of 978-2 are not available
   refund-payment: COMPENSATED
   cancel-order: COMPENSATED
   => order 102 is CANCELLED | payment: REFUNDED | stock of 978-2: 3

3. Six copies of 978-1 at 450 (card limit 2000)
   create-order: DONE
   charge-payment: FAILED - payment declined: 2700 is over the card limit of 2000
   cancel-order: COMPENSATED
   => order 103 is CANCELLED | payment: none | stock of 978-1: 8
Explanation of the output
  • The indented lines are the rows of saga_log for each order, and the line starting with => is the final state in the tables of the three services.
  • Order 101 - all steps are DONE. The order is CONFIRMED, the payment CHARGED and the stock of the book went from 10 to 8.
  • Order 102 - the payment of 500 was charged and committed. Then the stock step failed. The orchestrator ran the compensation of the one completed step, refund-payment, and then cancelled the order. The payment row is still there, now REFUNDED: a compensation leaves a trace where a rollback would leave none. The stock of 3 is untouched.
  • Order 103 - the first step after creating the order failed, so no step had completed and there was nothing to compensate except the order itself. No payment row exists.
  • The compensation release-stock never ran. Reserving stock is the last step that can fail, so in this saga nothing ever fails after it. It is defined all the same: the day a step is added after it, it will be needed.
  • In all three cases the system ended in a consistent state without any transaction that covered more than one service.

Choreography: no conductor

In choreography there is no orchestrator. Each service does its step and publishes an event saying what happened. Other services listen for the events that concern them and react. The saga is not written down anywhere; it emerges from who listens to what. Saga choreography - the order service publishes OrderCreated; the payment service reacts and publishes PaymentCompleted; the inventory service reacts and publishes StockReserved, which the order service hears to confirm the order; on failure the inventory service publishes StockRejected, the payment service refunds and publishes PaymentRefunded, and the order service cancels the order; all events travel through a message broker The same three services are reused. The events are records -
choreography/Events.java
//Events.java
package com.javabrahman.cloud.saga.choreography;

public final class Events {

  private Events() {}

  public record OrderCreated(long orderId, String isbn, int quantity, int amount) {}
  public record PaymentCompleted(long orderId, String isbn, int quantity) {}
  public record PaymentFailed(long orderId, String reason) {}
  public record StockReserved(long orderId) {}
  public record StockRejected(long orderId, String reason) {}
  public record PaymentRefunded(long orderId) {}
}
Each service gets a listener which turns incoming events into calls on the service and results into new events. Between real services the events would travel through a message broker. Here they are Spring application eventsClick to read tutorial: Application Events in Spring with @EventListener inside one application, which shows the flow of the saga without the infrastructure -
choreography/PaymentEvents.java
//PaymentEvents.java
package com.javabrahman.cloud.saga.choreography;

import com.javabrahman.cloud.saga.services.PaymentService;
import com.javabrahman.cloud.saga.services.SagaStepException;
import org.springframework.context.ApplicationEventPublisher;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;

@Component
public class PaymentEvents {

  private final PaymentService payments;
  private final ApplicationEventPublisher events;

  PaymentEvents(PaymentService payments, ApplicationEventPublisher events) {
    this.payments = payments;
    this.events = events;
  }

  @EventListener
  void on(Events.OrderCreated event) {
    try {
      payments.charge(event.orderId(), event.amount());
      System.out.println("   [payment]   heard OrderCreated     -> charged " + event.amount());
      events.publishEvent(new Events.PaymentCompleted(event.orderId(), event.isbn(), event.quantity()));
    } catch (SagaStepException e) {
      System.out.println("   [payment]   heard OrderCreated     -> " + e.getMessage());
      events.publishEvent(new Events.PaymentFailed(event.orderId(), e.getMessage()));
    }
  }

  // the compensation: the payment service itself knows that rejected stock means a refund
  @EventListener
  void on(Events.StockRejected event) {
    payments.refund(event.orderId());
    System.out.println("   [payment]   heard StockRejected    -> refunded");
    events.publishEvent(new Events.PaymentRefunded(event.orderId()));
  }
}
choreography/InventoryEvents.java
//InventoryEvents.java
package com.javabrahman.cloud.saga.choreography;

import com.javabrahman.cloud.saga.services.InventoryService;
import com.javabrahman.cloud.saga.services.SagaStepException;
import org.springframework.context.ApplicationEventPublisher;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;

@Component
public class InventoryEvents {

  private final InventoryService inventory;
  private final ApplicationEventPublisher events;

  InventoryEvents(InventoryService inventory, ApplicationEventPublisher events) {
    this.inventory = inventory;
    this.events = events;
  }

  @EventListener
  void on(Events.PaymentCompleted event) {
    try {
      inventory.reserve(event.orderId(), event.isbn(), event.quantity());
      System.out.println("   [inventory] heard PaymentCompleted -> reserved " + event.quantity() + " copies");
      events.publishEvent(new Events.StockReserved(event.orderId()));
    } catch (SagaStepException e) {
      System.out.println("   [inventory] heard PaymentCompleted -> " + e.getMessage());
      events.publishEvent(new Events.StockRejected(event.orderId(), e.getMessage()));
    }
  }
}
choreography/OrderEvents.java
//OrderEvents.java
package com.javabrahman.cloud.saga.choreography;

import com.javabrahman.cloud.saga.services.OrderService;
import org.springframework.context.ApplicationEventPublisher;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;

@Component
public class OrderEvents {

  private final OrderService orders;
  private final ApplicationEventPublisher events;

  OrderEvents(OrderService orders, ApplicationEventPublisher events) {
    this.orders = orders;
    this.events = events;
  }

  // the saga starts here: a local transaction and an event
  public long placeOrder(String isbn, int quantity, int unitPrice) {
    long orderId = orders.create(isbn, quantity, quantity * unitPrice);
    System.out.println("   [order]     created order " + orderId + " as PENDING");
    events.publishEvent(new Events.OrderCreated(orderId, isbn, quantity, quantity * unitPrice));
    return orderId;
  }

  @EventListener
  void on(Events.StockReserved event) {
    orders.confirm(event.orderId());
    System.out.println("   [order]     heard StockReserved    -> confirmed");
  }

  @EventListener
  void on(Events.PaymentFailed event) {
    orders.cancel(event.orderId());
    System.out.println("   [order]     heard PaymentFailed    -> cancelled");
  }

  @EventListener
  void on(Events.PaymentRefunded event) {
    orders.cancel(event.orderId());
    System.out.println("   [order]     heard PaymentRefunded  -> cancelled");
  }
}
choreography/ChoreographyApp.java
//ChoreographyApp.java
package com.javabrahman.cloud.saga.choreography;

import com.javabrahman.cloud.saga.services.ShopState;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;

@SpringBootApplication(scanBasePackages = {"com.javabrahman.cloud.saga.services", "com.javabrahman.cloud.saga.choreography"})
public class ChoreographyApp {

  @Bean
  CommandLineRunner demo(OrderEvents orders, ShopState state) {
    return args -> {
      run(orders, state, "1. Two copies of 978-1 at 450", "978-1", 2, 450);
      run(orders, state, "2. Five copies of 978-2 at 100 (three in stock)", "978-2", 5, 100);
      run(orders, state, "3. Six copies of 978-1 at 450 (card limit 2000)", "978-1", 6, 450);
    };
  }

  private void run(OrderEvents orders, ShopState state, String title, String isbn, int quantity, int unitPrice) {
    System.out.println(title);
    long orderId = orders.placeOrder(isbn, quantity, unitPrice);
    System.out.println("   => " + state.of(orderId, isbn));
    System.out.println();
  }

  public static void main(String[] args) {
    SpringApplication.run(ChoreographyApp.class, args);
  }
}
 OUTPUT of the choreographed saga
1. Two copies of 978-1 at 450
   [order]     created order 101 as PENDING
   [payment]   heard OrderCreated     -> charged 900
   [inventory] heard PaymentCompleted -> reserved 2 copies
   [order]     heard StockReserved    -> confirmed
   => order 101 is CONFIRMED | payment: CHARGED | stock of 978-1: 8

2. Five copies of 978-2 at 100 (three in stock)
   [order]     created order 102 as PENDING
   [payment]   heard OrderCreated     -> charged 500
   [inventory] heard PaymentCompleted -> out of stock: 5 copies of 978-2 are not available
   [payment]   heard StockRejected    -> refunded
   [order]     heard PaymentRefunded  -> cancelled
   => order 102 is CANCELLED | payment: REFUNDED | stock of 978-2: 3

3. Six copies of 978-1 at 450 (card limit 2000)
   [order]     created order 103 as PENDING
   [payment]   heard OrderCreated     -> payment declined: 2700 is over the card limit of 2000
   [order]     heard PaymentFailed    -> cancelled
   => order 103 is CANCELLED | payment: none | stock of 978-1: 8
Explanation of the output
  • Each line is printed by the listener of one service, named in brackets, and shows the event it heard and what it did.
  • Order 101 - the chain of the happy path: OrderCreated, PaymentCompleted, StockReserved. Three services acted, and none of them called another.
  • Order 102 - the inventory service could not reserve and published StockRejected. The payment service heard it and refunded; its PaymentRefunded event made the order service cancel. The compensation ran backwards through the services as a chain of events.
  • Order 103 - the payment service published PaymentFailed, the order service cancelled, and the inventory service never heard of the order.
  • The final states are the same as with the orchestrator. What differs is where the knowledge sits: "rejected stock means a refund" is now a decision of the payment service, and "a refund means cancel" one of the order service.

Orchestration or choreography?

Orchestration compared with choreography - orchestration has the flow in one place, is easy to follow and to monitor, but adds a central component that knows every service; choreography has no central component and loose coupling, but the flow is spread over the listeners of all services and hard to see as a whole; choreography suits two or three steps, orchestration suits longer flows
  • Orchestration keeps the whole flow in one class that can be read, tested and monitored, and the saga log shows where every order stands. The cost is one more component, on which every such order depends and which knows all the services.
  • Choreography needs no central component, and a new service can join by listening, without changes to the others. But the flow exists nowhere as a whole. To find out what happens after PaymentCompleted, one must search every service for listeners, and a chain of events that loops back is easy to create by accident.
  • A common rule: choreography for sagas of two or three steps, orchestration for anything longer or anything that has to be explained to an auditor.

What a production saga needs

The example leaves out the hard parts, and they should be known before a saga is built -
  • Surviving a crash - if the orchestrator stops between two steps, the saga must be continued or compensated after the restart. The saga log exists for this: it must be written durably, and a recovery job must pick up unfinished sagas.
  • Idempotent steps - after a timeout nobody knows whether a step was carried out, so it is sent again. A step and a compensation must therefore be safe to run twice: charging the same order a second time must not take the money twice.
  • Compensations must succeed - a compensation that fails cannot itself be compensated. It is retried until it works, and if it never does, a person must be told.
  • Database and message together - "update my table and publish an event" are two actions, and a crash between them loses the event. The transactional outbox solves it: the event is written to a table in the same local transaction, and a separate process publishes it from there.
  • Steps that cannot be undone - an email that was sent, a parcel that left. Such steps are placed last, after everything that can fail.
  • Tools - for long or many sagas, a workflow engine such as Temporal, Camunda or Axon provides the durable log, the retries and the recovery, and is usually a better choice than writing them.
And one more piece of advice: the best saga is the one that is not needed. If two pieces of data must always change together, that is a strong hint that they belong in the same service, as discussed in when and why to use microservicesClick to read tutorial: Microservices with Spring: When and Why.

Summary

In this tutorial we saw why a business operation across several services cannot be one transaction, and how a saga replaces it with a sequence of local transactions and compensating actions. An orchestrated saga ran an order through payment and stock, logged each step, and on a failure refunded and cancelled in reverse order. The same flow was then built as choreography, with each service reacting to the events of the others. Both ended in the same consistent states. Orchestration puts the flow in one visible place; choreography spreads it and couples less. Either way a real saga needs durable state, idempotent steps and compensations that are retried until they succeed.

Saga Pattern for Distributed Transactions - JavaBrahman