Spring Boot Testing Best Practices and Faster Test Suites

Introduction

This tutorial is about keeping a Spring Boot test suite fast and reliable as it grows. We will see Spring's test context cache at work, by counting the application contexts that six slightly different test classes start, and then cover the same checks with a single context. We will see the price of @DirtiesContext, run slow tests in parallel with JUnit's parallel execution, and replace Thread.sleep() in tests of asynchronous code with Awaitility. A list of good practices closes the tutorial, and the module.

Where the time goes

In most Spring Boot test suites, the test methods themselves are fast; the time goes into starting application contexts. Starting one means scanning components, running auto-configuration, creating hundreds of beans, often starting an embedded database and building a JPA entity manager factory. Spring's test framework therefore keeps every context it starts in a cache, for the whole run of the test JVM, and gives it to every later test class whose configuration is the same. The key of that cache is the test class's whole configuration, so a test class that differs in anything - one more @MockitoBean, one more property - needs a context of its own - The test context cache - six test classes started four application contexts, because a mock bean, a property and @DirtiesContext each forced a context of its own

The test context cache - six test classes started four application contexts, because a mock bean, a property and @DirtiesContext each forced a context of its own

The diagram shows the result of the first test run below. The bigger the application, the more each extra context costs.

The application

A book service reads prices from a JPA repository, applies a discount from the property shop.discount, and converts prices to US dollars through a client for an external exchange-rate service -

src/main/java/.../Book.java

//Book.java
package com.javabrahman.testing.practices;

import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.Id;

@Entity
public class Book {
  @Id
  @GeneratedValue
  private Long id;
  private String title;
  private int price;

  protected Book() {
  }

  public Book(String title, int price) {
    this.title = title;
    this.price = price;
  }

  public Long getId() {
    return id;
  }

  public String getTitle() {
    return title;
  }

  public int getPrice() {
    return price;
  }
}

src/main/java/.../BookRepository.java

//BookRepository.java
package com.javabrahman.testing.practices;

import org.springframework.data.jpa.repository.JpaRepository;

public interface BookRepository extends JpaRepository<Book, Long> {
}

src/main/java/.../ExchangeRateClient.java - calls an external service

//ExchangeRateClient.java
package com.javabrahman.testing.practices;

import org.springframework.stereotype.Component;

@Component
public class ExchangeRateClient {

  public double inrPerUsd() {
    throw new IllegalStateException("no network access in this example");
  }
}

src/main/java/.../BookService.java

//BookService.java
package com.javabrahman.testing.practices;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

@Service
public class BookService {
  private final BookRepository books;
  private final ExchangeRateClient rates;
  private final int discountPercent;

  public BookService(BookRepository books, ExchangeRateClient rates, @Value("${shop.discount:0}") int discountPercent) {
    this.books = books;
    this.rates = rates;
    this.discountPercent = discountPercent;
  }

  public int priceInr(long id) {
    int price = books.findById(id).orElseThrow().getPrice();
    return price - price * discountPercent / 100;
  }

  public double priceUsd(long id) {
    return Math.round(priceInr(id) / rates.inrPerUsd() * 100) / 100.0;
  }
}

src/main/resources/application.properties

spring.jpa.open-in-view=false

To make the contexts visible, the tests have a small helper: an application listener that prints a line each time Spring Boot starts an application context. It is registered in a test-only spring.factories file, so that no test class has to import it - importing it would itself be part of the test configuration -

src/test/java/.../ContextStartCounter.java - counts the application contexts a test run starts

//ContextStartCounter.java
package com.javabrahman.testing.practices;

import java.util.concurrent.atomic.AtomicInteger;
import org.springframework.boot.context.event.ApplicationStartedEvent;
import org.springframework.context.ApplicationListener;

// a test-only listener: says so whenever Spring Boot starts a new application context
public class ContextStartCounter implements ApplicationListener<ApplicationStartedEvent> {
  private static final AtomicInteger STARTED = new AtomicInteger();

  @Override
  public void onApplicationEvent(ApplicationStartedEvent event) {
    System.out.println("   >> application context #" + STARTED.incrementAndGet() + " started");
  }
}

src/test/resources/META-INF/spring.factories - registers the listener for the tests

org.springframework.context.ApplicationListener=com.javabrahman.testing.practices.ContextStartCounter

src/main/java/.../ShopApp.java

//ShopApp.java
package com.javabrahman.testing.practices;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class ShopApp {

  public static void main(String[] args) {
    SpringApplication.run(ShopApp.class, args);
  }
}

Six test classes, six configurations?

These test classes are typical of a suite that grew over time: each was written by someone who added what that one test needed. They print their names in @BeforeAll, which runs before Spring prepares the test instance, so that each counter line appears under the class that caused it. @Order fixes the order of the classes for this demonstration, with JUnit's ClassOrderer.OrderAnnotation switched on for the run -

src/test/java/.../PriceTest.java

//PriceTest.java
package com.javabrahman.testing.practices;

import static org.assertj.core.api.Assertions.assertThat;

import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

@Order(1)
@SpringBootTest
class PriceTest {
  @Autowired BookRepository books;
  @Autowired BookService service;

  @BeforeAll
  static void name() { System.out.println("PriceTest"); }

  @Test
  void fullPriceWithoutDiscount() {
    long id = books.save(new Book("Effective Java", 450)).getId();
    assertThat(service.priceInr(id)).isEqualTo(450);
  }
}

src/test/java/.../RepositoryTest.java

//RepositoryTest.java
package com.javabrahman.testing.practices;

import static org.assertj.core.api.Assertions.assertThat;

import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

@Order(2)
@SpringBootTest
class RepositoryTest {
  @Autowired BookRepository books;

  @BeforeAll
  static void name() { System.out.println("RepositoryTest"); }

  @Test
  void savesABook() {
    assertThat(books.save(new Book("Clean Code", 380)).getId()).isNotNull();
  }
}

src/test/java/.../UsdPriceTest.java - adds a @MockitoBean

//UsdPriceTest.java
package com.javabrahman.testing.practices;

import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.BDDMockito.given;

import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.bean.override.mockito.MockitoBean;

@Order(3)
@SpringBootTest
class UsdPriceTest {
  @Autowired BookRepository books;
  @Autowired BookService service;
  @MockitoBean ExchangeRateClient rates;

  @BeforeAll
  static void name() { System.out.println("UsdPriceTest (adds a @MockitoBean)"); }

  @Test
  void convertsToUsd() {
    given(rates.inrPerUsd()).willReturn(90.0);
    long id = books.save(new Book("Refactoring", 900)).getId();
    assertThat(service.priceUsd(id)).isEqualTo(10.0);
  }
}

src/test/java/.../DiscountTest.java - sets a property

//DiscountTest.java
package com.javabrahman.testing.practices;

import static org.assertj.core.api.Assertions.assertThat;

import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

@Order(4)
@SpringBootTest(properties = "shop.discount=20")
class DiscountTest {
  @Autowired BookRepository books;
  @Autowired BookService service;

  @BeforeAll
  static void name() { System.out.println("DiscountTest (sets a property)"); }

  @Test
  void twentyPercentOff() {
    long id = books.save(new Book("Domain-Driven Design", 500)).getId();
    assertThat(service.priceInr(id)).isEqualTo(400);
  }
}

src/test/java/.../DirtyTest.java - with @DirtiesContext

//DirtyTest.java
package com.javabrahman.testing.practices;

import static org.assertj.core.api.Assertions.assertThat;

import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.annotation.DirtiesContext;

@Order(5)
@SpringBootTest
@DirtiesContext
class DirtyTest {
  @Autowired BookRepository books;

  @BeforeAll
  static void name() { System.out.println("DirtyTest (@DirtiesContext)"); }

  @Test
  void deletesEverything() {
    books.deleteAll();
    assertThat(books.count()).isZero();
  }
}

src/test/java/.../AfterDirtyTest.java

//AfterDirtyTest.java
package com.javabrahman.testing.practices;

import static org.assertj.core.api.Assertions.assertThat;

import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

@Order(6)
@SpringBootTest
class AfterDirtyTest {
  @Autowired BookRepository books;

  @BeforeAll
  static void name() { System.out.println("AfterDirtyTest"); }

  @Test
  void countsBooks() {
    assertThat(books.count()).isZero();
  }
}

OUTPUT - six test classes

PriceTest
   >> application context #1 started
RepositoryTest
UsdPriceTest (adds a @MockitoBean)
   >> application context #2 started
DiscountTest (sets a property)
   >> application context #3 started
DirtyTest (@DirtiesContext)
AfterDirtyTest
   >> application context #4 started
╷
└─ JUnit Jupiter ✔
   ├─ PriceTest ✔
   │  └─ fullPriceWithoutDiscount() ✔
   ├─ RepositoryTest ✔
   │  └─ savesABook() ✔
   ├─ UsdPriceTest ✔
   │  └─ convertsToUsd() ✔
   ├─ DiscountTest ✔
   │  └─ twentyPercentOff() ✔
   ├─ DirtyTest ✔
   │  └─ deletesEverything() ✔
   └─ AfterDirtyTest ✔
      └─ countsBooks() ✔

[         7 containers found      ]
[         0 containers skipped    ]
[         7 containers started    ]
[         0 containers aborted    ]
[         7 containers successful ]
[         0 containers failed     ]
[         6 tests found           ]
[         0 tests skipped         ]
[         6 tests started         ]
[         0 tests aborted         ]
[         6 tests successful      ]
[         0 tests failed          ]

Explanation of the output

A cache hit: RepositoryTest printed no counter line. Its configuration equals PriceTest's, so it got context #1 from the cache.
Two misses: UsdPriceTest's @MockitoBean and DiscountTest's property each made a new configuration, and each started its own context, #2 and #3, although both tests use the same beans as the first two.
@DirtiesContext: DirtyTest reused context #1, and then, because of @DirtiesContext, Spring closed it and removed it from the cache. AfterDirtyTest, with the plain configuration again, had to start context #4, which came with a new embedded H2 database: Spring Boot gives the embedded database of every context a unique name, so contexts never share one.
Several contexts stay open: contexts #2 and #3 stayed in the cache until the end of the run, each with its own database and connection pool. The cache holds up to 32 contexts by default, set by the system property spring.test.context.cache.maxSize.

Running the tests with logging.level.org.springframework.test.context.cache=DEBUG makes Spring log the cache's statistics - its size, its hit count and its miss count - each time a test asks it for a context. A miss count close to the number of test classes is the sign of a suite that keeps starting contexts.

One configuration, one context

The fix is to make the integration tests share one configuration, in a base class that declares everything they need, including the mock of the external service. A test that needs a different property value, such as the discount, is often better as a unit test: the discount is plain arithmetic, and needs no Spring at all -

src/test/java/.../IntegrationTest.java - one configuration for all integration tests

//IntegrationTest.java
package com.javabrahman.testing.practices;

import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.bean.override.mockito.MockitoBean;

@SpringBootTest
public abstract class IntegrationTest {

  // the external service is mocked once, for every integration test
  @MockitoBean
  protected ExchangeRateClient rates;
}

src/test/java/.../CatalogIT.java

//CatalogIT.java
package com.javabrahman.testing.practices;

import static org.assertj.core.api.Assertions.assertThat;

import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;

class CatalogIT extends IntegrationTest {
  @Autowired BookRepository books;
  @Autowired BookService service;

  @BeforeAll
  static void name() { System.out.println("CatalogIT"); }

  @Test
  void savesABook() {
    assertThat(books.save(new Book("Clean Code", 380)).getId()).isNotNull();
  }

  @Test
  void fullPriceWithoutDiscount() {
    long id = books.save(new Book("Effective Java", 450)).getId();
    assertThat(service.priceInr(id)).isEqualTo(450);
  }
}

src/test/java/.../UsdPriceIT.java

//UsdPriceIT.java
package com.javabrahman.testing.practices;

import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.BDDMockito.given;

import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;

class UsdPriceIT extends IntegrationTest {
  @Autowired BookRepository books;
  @Autowired BookService service;

  @BeforeAll
  static void name() { System.out.println("UsdPriceIT"); }

  @Test
  void convertsToUsd() {
    given(rates.inrPerUsd()).willReturn(90.0);
    long id = books.save(new Book("Refactoring", 900)).getId();
    assertThat(service.priceUsd(id)).isEqualTo(10.0);
  }
}

src/test/java/.../DiscountUnitTest.java - no Spring needed

//DiscountUnitTest.java
package com.javabrahman.testing.practices;

import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.BDDMockito.given;
import static org.mockito.Mockito.mock;

import java.util.Optional;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;

class DiscountUnitTest {

  @BeforeAll
  static void name() { System.out.println("DiscountUnitTest"); }

  @Test
  void twentyPercentOff() {
    BookRepository books = mock(BookRepository.class);
    given(books.findById(7L)).willReturn(Optional.of(new Book("Domain-Driven Design", 500)));
    BookService service = new BookService(books, mock(ExchangeRateClient.class), 20);
    assertThat(service.priceInr(7)).isEqualTo(400);
  }
}

OUTPUT - the same checks with one shared configuration

CatalogIT
   >> application context #1 started
UsdPriceIT
DiscountUnitTest
╷
└─ JUnit Jupiter ✔
   ├─ CatalogIT ✔
   │  ├─ fullPriceWithoutDiscount() ✔
   │  └─ savesABook() ✔
   ├─ UsdPriceIT ✔
   │  └─ convertsToUsd() ✔
   └─ DiscountUnitTest ✔
      └─ twentyPercentOff() ✔

[         4 containers found      ]
[         0 containers skipped    ]
[         4 containers started    ]
[         0 containers aborted    ]
[         4 containers successful ]
[         0 containers failed     ]
[         4 tests found           ]
[         0 tests skipped         ]
[         4 tests started         ]
[         0 tests aborted         ]
[         4 tests successful      ]
[         0 tests failed          ]

One context was started for both integration test classes, and the unit test started none. A @MockitoBean declared in the base class costs nothing in tests that do not use it: Mockito resets it after every test, and an unstubbed mock simply returns defaults. The names ending in IT follow a common convention: Maven's Failsafe plugin runs them as integration tests, separately from the unit tests that Surefire runs, so that a quick mvn test runs only the unit tests.

Running tests in parallel

JUnit runs the tests of a JVM one after the other by default. Its parallel execution runs them on several threads, switched on with configuration parameters - in a project, in src/test/resources/junit-platform.properties. Four tests of half a second each show the difference -

src/test/java/.../SlowTests.java - four tests that wait

//SlowTests.java
package com.javabrahman.testing.practices;

import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;

class SlowTests {
  private static final Set<String> threads = ConcurrentHashMap.newKeySet();
  private static long start;

  @BeforeAll
  static void start() {
    start = System.nanoTime();
  }

  @Test void first() throws InterruptedException { work(); }
  @Test void second() throws InterruptedException { work(); }
  @Test void third() throws InterruptedException { work(); }
  @Test void fourth() throws InterruptedException { work(); }

  private static void work() throws InterruptedException {
    threads.add(Thread.currentThread().getName());
    Thread.sleep(500);                       // stands for half a second of real work
  }

  @AfterAll
  static void report() {
    long millis = (System.nanoTime() - start) / 1_000_000;
    System.out.println("   4 tests of 0.5 s each ran on " + threads.size() + " thread(s), in "
        + (millis < 1000 ? "less than 1 second" : "2 seconds or more"));
  }
}

OUTPUT - one thread (the default)

   4 tests of 0.5 s each ran on 1 thread(s), in 2 seconds or more
╷
└─ JUnit Jupiter ✔
   └─ SlowTests ✔
      ├─ fourth() ✔
      ├─ second() ✔
      ├─ first() ✔
      └─ third() ✔

[         2 containers found      ]
[         0 containers skipped    ]
[         2 containers started    ]
[         0 containers aborted    ]
[         2 containers successful ]
[         0 containers failed     ]
[         4 tests found           ]
[         0 tests skipped         ]
[         4 tests started         ]
[         0 tests aborted         ]
[         4 tests successful      ]
[         0 tests failed          ]

OUTPUT - with parallel execution switched on

   4 tests of 0.5 s each ran on 4 thread(s), in less than 1 second
╷
└─ JUnit Jupiter ✔
   └─ SlowTests ✔
      ├─ second() ✔
      ├─ third() ✔
      ├─ fourth() ✔
      └─ first() ✔

[         2 containers found      ]
[         0 containers skipped    ]
[         2 containers started    ]
[         0 containers aborted    ]
[         2 containers successful ]
[         0 containers failed     ]
[         4 tests found           ]
[         0 tests skipped         ]
[         4 tests started         ]
[         0 tests aborted         ]
[         4 tests successful      ]
[         0 tests failed          ]

src/test/resources/junit-platform.properties - the settings of the second run

junit.jupiter.execution.parallel.enabled=true
junit.jupiter.execution.parallel.mode.default=concurrent
junit.jupiter.execution.parallel.config.strategy=fixed
junit.jupiter.execution.parallel.config.fixed.parallelism=4

The same four tests took over two seconds on one thread, and under a second on four. mode.default=concurrent runs the methods of a class concurrently; the dynamic strategy, the default, sizes the thread pool from the number of processors instead of a fixed number.

Parallel tests must not share changeable state. Tests that use the same Spring context share its beans and its database, so two of them saving and counting books at the same time can see each other's data. @Execution(ExecutionMode.SAME_THREAD) keeps a class on one thread, and @ResourceLock("books") stops tests that use the same resource from running at the same time. Many teams run unit tests in parallel and Spring integration tests one after the other, or in parallel JVMs, through Surefire's forkCount, where each JVM has its own context cache.

Waiting without Thread.sleep()

A test of asynchronous code - an @Async method, a message listener, a scheduled job - must wait for the result. Thread.sleep(2000) waits two seconds even when the work took 300 ms, and still fails on a slow build machine. Awaitility, part of Spring Boot's test starter, polls instead, and returns as soon as the condition holds, failing only after the given maximum -

src/test/java/.../ReportJobTest.java - waiting for asynchronous work

//ReportJobTest.java
package com.javabrahman.testing.practices;

import static org.awaitility.Awaitility.await;

import java.time.Duration;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.atomic.AtomicBoolean;
import org.junit.jupiter.api.Test;

class ReportJobTest {

  @Test
  void reportIsReadySoon() {
    AtomicBoolean ready = new AtomicBoolean();
    CompletableFuture.runAsync(() -> {
      sleep(300);                            // the job takes a while, in another thread
      ready.set(true);
    });

    long start = System.nanoTime();
    await().atMost(Duration.ofSeconds(2)).untilTrue(ready);   // polls, and returns as soon as it is true
    long waited = (System.nanoTime() - start) / 1_000_000;
    System.out.println("   ready after " + (waited < 1000 ? "less than 1 second" : "1 second or more"));
  }

  private static void sleep(long millis) {
    try {
      Thread.sleep(millis);
    } catch (InterruptedException e) {
      Thread.currentThread().interrupt();
    }
  }
}

OUTPUT - Awaitility

   ready after less than 1 second
╷
└─ JUnit Jupiter ✔
   └─ ReportJobTest ✔
      └─ reportIsReadySoon() ✔

[         2 containers found      ]
[         0 containers skipped    ]
[         2 containers started    ]
[         0 containers aborted    ]
[         2 containers successful ]
[         0 containers failed     ]
[         1 tests found           ]
[         0 tests skipped         ]
[         1 tests started         ]
[         0 tests aborted         ]
[         1 tests successful      ]
[         0 tests failed          ]

The test waited for the job, which took about 300 ms, and not for the two seconds allowed. Besides untilTrue(), Awaitility offers until(() -> ...) for any condition and untilAsserted(() -> ...) for an AssertJ assertion that must eventually pass.

Good practice for Spring Boot tests

Keep the pyramid: many unit tests, fewer slice tests, few full @SpringBootTests, and a handful of end-to-end tests, as described in the test pyramid tutorialClick to read tutorial: Testing in Spring Boot: The Test Pyramid and @SpringBootTest.
Prefer slices to full contexts for a single layer: @WebMvcTest and @DataJpaTest start a fraction of the beans, as the bean counts in the slice tests tutorialClick to read tutorial: Spring Boot Slice Tests: @WebMvcTest, @DataJpaTest and More showed.
Share configurations: base classes or meta-annotations for the common setups, so that the context cache can do its work. Every @MockitoBean, property, profile or @Import that differs from one class to the next costs a context.
Avoid @DirtiesContext: clean up the data a test changed instead - with @Transactional test methods, which are rolled back, or an explicit cleanup - and keep @DirtiesContext for a test that truly breaks a context.
Know when rollback does not help: with webEnvironment = RANDOM_PORT, the request runs on a server thread in its own transaction, so a test's @Transactional does not roll it back.
One behaviour per test, named after it: twentyPercentOff() says what broke when it fails; test1() does not.
Make tests independent of order: a test must pass alone and in any order. The @Order used in this tutorial is for the demonstration only.
No sleeps, no real clocks, no network: Awaitility for asynchronous work, an injected Clock for time, and mocks, stubs or WireMock for external services.

Summary

In this tutorial we made a test suite faster. A counter of application context starts showed the test context cache at work: six test classes, differing in one mock bean, one property and one @DirtiesContext, started four contexts, and the same checks, with a shared base class and a unit test, needed one. JUnit's parallel execution ran four slow tests on four threads in a quarter of the time, and Awaitility replaced a fixed sleep with a wait that ends as soon as the work is done. With this, the testing module of the Spring series is complete: from the test pyramid and slices, through mocks, MockMvc and security tests, to contracts between services and the speed of the whole suite.

Spring Testing Tutorials
Contract Testing with Spring Cloud Contract Testing in Spring Boot: The Test Pyramid and @SpringBootTest Spring Boot Slice Tests: @WebMvcTest, @DataJpaTest and More Mocking Beans in Spring Boot Tests with @MockitoBean Testing REST APIs with MockMvc and MockMvcTester Testing Spring Security: @WithMockUser, CSRF and JWT