Taming Flaky Tests with Docker Containers

The Problem with Flaky Tests

Flaky tests compromise software integrity and slow down development velocity. They can be caused by various factors, including randomness, race conditions, or dependencies on external services. When tests are non-deterministic, it’s challenging to identify the root cause of failures.

Why Use Docker Containers?

Docker containers provide a consistent and isolated environment for testing, eliminating the variability that leads to flaky tests. By using containers, we can:

• Isolate the application’s runtime environment from the host operating system
• Minimize shared resource usage
• Ensure tests are properly isolated

Testing a Node.js Application

Let’s test an Express.js-powered Node.js app that stores and reads job listings. We’ll use Postgres for persistence and Knex.js for managing migrations and seeds.

Setting Up the Test Environment

We’ll create a Dockerfile and a docker-compose.yml manifest to define our test environment.

FROM node:14

WORKDIR /app

COPY package*.json./

RUN npm install

COPY..

RUN npm run build

EXPOSE 3000

CMD ["npm", "start"]

version: '3'
services:
  simple-job-board:
    build:.
    ports:
      - "3000:3000"
    depends_on:
      - db
    environment:
      - DATABASE_URL=postgres://user:password@db:5432/simple-job-board
  db:
    image: postgres
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=simple-job-board

Running Tests in Docker Containers

With our test environment set up, we can run our tests using docker-compose run test. This ensures that our database service spins up before our tests start running. Our test results will pass with green checkmarks, indicating that our tests are running reliably in the Docker containers.

Reducing Flakiness

By using Docker containers and test isolation, we’ve greatly reduced the chances of encountering flaky tests. We’ve created new connections and spun up new databases for each test run, guaranteeing a clean slate for each run. We’ve also eliminated state contamination as a source of flakiness.

Remember that test flakiness is multi-variate in nature, and there are other sources of flakiness, such as:

• async behavior
• concurrency issues
• network interactions