Containerizing a Microservices Architecture with Docker and Redpanda

Introduction

Hello there!

I'm Bobby, a Docker Captain and the author of the free Introduction to Docker eBook. In this guide, we'll dive deep into containerizing a microservices architecture using Docker and Redpanda, a Kafka-compatible event streaming platform.

As we progress through 2024, microservices and event-driven architectures continue to be at the leading approach of modern application development, and Docker remains an indispensable tool for deploying and managing these complex systems.

Prerequisites

Before we go any further, make sure that you have:

• Docker and Docker Compose installed on your system
• Basic knowledge of Docker, microservices concepts, and event streaming

If you're new to Docker or need a refresher, I highly recommend checking out my free Introduction to Docker eBook. It covers all the fundamentals you'll need to follow along with this guide.

Our Example Architecture

For this article, we'll build a simple e-commerce platform with the following microservices:

1. Product Service (Go)
2. User Service (Python)
3. Order Service (Node.js)
4. API Gateway (Nginx)
5. Event Streaming Platform (Redpanda)
6. Database (PostgreSQL)

This architecture demonstrates a typical microservices setup, with each service responsible for a specific domain and communicating through both REST APIs and event streaming.

An example diagram of the architecture is shown below. The client apps (web, mobile, etc.) interact with the API Gateway, which routes requests to the appropriate microservices. The services communicate with a PostgreSQL database for persistent storage and with Redpanda for event streaming.

                  +-------------------+
                  |    Client Apps    |
                  | (Web, Mobile, etc)|
                  +---------+---------+
                            |
                            | HTTP/HTTPS
                            |
                  +---------v---------+
                  |                   |
                  |    API Gateway    |
                  |     (Nginx)       |
                  |                   |
                  +----+------+-------+
                       |      |
           +-----------+      +-----------+
           |                              |
     +-----v------+  +-----------+  +-----v------+
     |            |  |           |  |            |
     |  Product   |  |   User    |  |   Order    |
     |  Service   |  |  Service  |  |  Service   |
     |   (Go)     |  | (Python)  |  | (Node.js)  |
     |            |  |           |  |            |
     +-----+------+  +-----+-----+  +-----+------+
           |              |              |
           |              |              |
    +------v--------------v--------------v------+
    |                                           |
    |              PostgreSQL                   |
    |              Database                     |
    |                                           |
    +-------------------------------------------+
                       |
                       | (Event Streaming)
                       |
               +-------v-------+
               |               |
               |   Redpanda    |
               |               |
               +-------+-------+
                       |
                       | (External Systems)
                       |
               +-------v-------+
               | Notifications |
               |   Analytics   |
               | Materialize   |
               | Other Services|
               +---------------+

Step 1: Dockerizing Individual Services

Let's start by creating Dockerfiles for each service and include some basic code examples. We'll go through each service, explaining the Dockerfile and the code in detail.

Product Service (Go)

Dockerfile:

FROM golang:1.21-alpine AS builder
WORKDIR /app
COPY . .
RUN go build -o main .

FROM alpine:3.18
COPY --from=builder /app/main /app/main
CMD ["/app/main"]

This Dockerfile uses a multi-stage build:

1. The first stage uses the golang:1.21-alpine image to build our Go application.
2. The second stage uses a minimal alpine:3.18 image and copies only the compiled binary from the first stage.
3. This approach significantly reduces the final image size by excluding all build tools and intermediate files.

Basic Go code (main.go):

package main

import (
    "encoding/json"
    "log"
    "net/http"
)

type Product struct {
    ID    string  `json:"id"`
    Name  string  `json:"name"`
    Price float64 `json:"price"`
}

func main() {
    http.HandleFunc("/products", func(w http.ResponseWriter, r *http.Request) {
        products := []Product{
            {ID: "1", Name: "Product 1", Price: 19.99},
            {ID: "2", Name: "Product 2", Price: 29.99},
        }
        json.NewEncoder(w).Encode(products)
    })

    log.Fatal(http.ListenAndServe(":8080", nil))
}

This Go code:

1. Defines a simple Product struct.
2. Creates an HTTP server that listens on port 8080.
3. Implements a /products endpoint that returns a JSON array of product data.
4. In a real-world scenario, this would interact with a database instead of using hardcoded data but for simplicity, we're using hardcoded data.

User Service (Python)

Dockerfile:

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "app.py"]

This Dockerfile:

1. Uses the official Python 3.11 slim image as the base.
2. Sets the working directory to /app.
3. Copies and installs the Python dependencies first (for better caching).
4. Copies the rest of the application code.
5. Specifies the command to run the Python application.

Basic Python code (app.py):

from flask import Flask, jsonify

app = Flask(__name__)

@app.route('/users')
def get_users():
    users = [
        {"id": "1", "name": "John Doe"},
        {"id": "2", "name": "Jane Smith"}
    ]
    return jsonify(users)

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8081)

This Python code:

1. Uses Flask to create a simple web server.
2. Defines a /users endpoint that returns a JSON array of user data.
3. Runs the server on port 8081 and binds to all available network interfaces (0.0.0.0).
4. Like the product service, in a real-world scenario, this would interact with a database.

Order Service (Node.js)

Dockerfile:

FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
CMD ["node", "server.js"]

This Dockerfile:

1. Uses the official Node.js 20 Alpine image as the base.
2. Sets the working directory to /app.
3. Copies package.json and package-lock.json files first.
4. Installs production dependencies using npm ci for faster, more reliable builds.
5. Copies the rest of the application code.
6. Specifies the command to run the Node.js application.

Basic Node.js code (server.js):

const express = require('express');
const { Kafka } = require('kafkajs');

const app = express();
const kafka = new Kafka({
  clientId: 'order-service',
  brokers: ['redpanda:9092']
});

const producer = kafka.producer();

app.use(express.json());

app.post('/orders', async (req, res) => {
  const order = req.body;
  await producer.connect();
  await producer.send({
    topic: 'orders',
    messages: [{ value: JSON.stringify(order) }],
  });
  await producer.disconnect();
  res.status(201).json({ message: 'Order created' });
});

app.listen(8082, () => console.log('Order service listening on port 8082'));

This Node.js code:

1. Uses Express to create a web server.
2. Configures a Kafka producer to connect to our Redpanda service.
3. Implements a POST /orders endpoint that:
   • Receives order data in the request body.
   • Sends the order data to a Kafka topic named 'orders'.
   • Responds with a success message.
4. This service demonstrates how to integrate with Redpanda for event streaming.

Step 2: Creating the Docker Compose File

Now, let's create a compose.yaml file to orchestrate our microservices. We'll go through each section of the Compose file in detail:

version: '3.9'

services:
  product-service:
    build: ./product-service
    ports:
      - "8080:8080"
    environment:
      - DB_HOST=postgres
    depends_on:
      - postgres

  user-service:
    build: ./user-service
    ports:
      - "8081:8081"
    environment:
      - DB_HOST=postgres
    depends_on:
      - postgres

  order-service:
    build: ./order-service
    ports:
      - "8082:8082"
    environment:
      - DB_HOST=postgres
      - REDPANDA_HOST=redpanda
    depends_on:
      - postgres
      - redpanda

  api-gateway:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - product-service
      - user-service
      - order-service

  postgres:
    image: postgres:13-alpine
    environment:
      POSTGRES_PASSWORD: mysecretpassword
    volumes:
      - pgdata:/var/lib/postgresql/data

  redpanda:
    container_name: redpanda
    image: docker.vectorized.io/vectorized/redpanda:v23.3.5
    depends_on:
      - postgres
    command:
      - redpanda start
      - --overprovisioned
      - --smp 1
      - --memory 1G
      - --reserve-memory 0M
      - --node-id 0
      - --check=false
      - --kafka-addr 0.0.0.0:9092
      - --advertise-kafka-addr redpanda:9092
      - --pandaproxy-addr 0.0.0.0:8082
      - --advertise-pandaproxy-addr redpanda:8082
      - --set redpanda.enable_transactions=true
      - --set redpanda.enable_idempotence=true
      - --set redpanda.auto_create_topics_enabled=true
      - --set redpanda.default_topic_partitions=1
    ports:
      - 9092:9092
      - 8081:8081
      - 8082:8082
    healthcheck:
      test: curl -f localhost:9644/v1/status/ready
      interval: 1s
      start_period: 30s

volumes:
  pgdata:

Let's break down the key components of this Compose file:

1. Version: We're using version 3.9 of the Compose file format, which is compatible with Docker Engine 19.03.0+.

2. Services:

   • Each service is defined under the services key.
   • For our custom services (product, user, order), we use the build directive to specify the location of the Dockerfile.
   • We map ports from the container to the host for each service.
   • Environment variables are set using the environment key.
   • Dependencies between services are specified using depends_on.

3. API Gateway:

   • Uses the official Nginx Alpine image.
   • Mounts a custom Nginx configuration file.
   • Depends on all other services to ensure they're running before the gateway starts.

4. PostgreSQL:

   • Uses the official PostgreSQL Alpine image.
   • Sets a password using an environment variable.
   • Uses a named volume for data persistence.

5. Redpanda:

   • Uses the official Redpanda image.
   • Configures various Redpanda-specific options.
   • Exposes necessary ports for Kafka protocol and admin interface.
   • Includes a health check to ensure the service is ready.

6. Volumes:

   • Defines a named volume pgdata for PostgreSQL data persistence. This volume is automatically created by Docker and will persist even if the container is removed or recreated.

Step 3: Configuring the API Gateway

The API Gateway uses Nginx to route requests to the appropriate microservices. Here's a detailed explanation of the nginx.conf file:

events {
    worker_connections 1024;
}

http {
    upstream product-service {
        server product-service:8080;
    }

    upstream user-service {
        server user-service:8081;
    }

    upstream order-service {
        server order-service:8082;
    }

    server {
        listen 80;

        location /products {
            proxy_pass http://product-service;
        }

        location /users {
            proxy_pass http://user-service;
        }

        location /orders {
            proxy_pass http://order-service;
        }
    }
}

This Nginx configuration:

1. Defines upstream servers for each of our microservices.
2. Creates a server block that listens on port 80.
3. Uses location blocks to route requests to the appropriate service based on the URL path.
4. This setup allows us to access all our microservices through a single entry point.

Step 4: Running the Microservices

To start all services, use the following command:

docker compose up -d

This command:

1. Builds any images that aren't cached (on the first run or when changes are made).
2. Creates and starts containers for all services defined in the Compose file.
3. The -d flag runs the containers in detached mode (in the background).

To view the logs of all running containers:

docker compose logs -f

The -f flag "follows" the log output, showing logs in real-time.

To stop all services:

docker compose down

This stops and removes all containers defined in the Compose file. Add the -v flag to also remove volumes.

Step 5: Testing the Services

You can test the services using curl or a tool like Postman. Here are some example curl commands:

1. Get products:

   curl http://localhost/products
   

   This should return a JSON array of products.

2. Get users:

   curl http://localhost/users
   

   This should return a JSON array of users.

3. Create an order:

   curl -X POST http://localhost/orders -H "Content-Type: application/json" -d '{"productId": "1", "userId": "1", "quantity": 2}'
   

   This should return a success message and send the order data to Redpanda.

Step 6: Scaling Services

One of the advantages of this architecture is the ability to scale services independently. To scale a service, use:

docker compose up -d --scale product-service=3

This command:

1. Increases the number of product-service containers to 3.
2. The API Gateway (Nginx) will automatically load balance requests between these instances.

You can verify the running containers with:

docker compose ps

Step 7: Monitoring and Logging

For a production environment, you'd want to add monitoring and centralized logging. Here's a basic example of adding Prometheus for monitoring:

Add this to your compose.yaml:

  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

Create a prometheus.yml file:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'redpanda'
    static_configs:
      - targets: ['redpanda:9644']

This setup will allow Prometheus to scrape metrics from itself and Redpanda.

Conclusion

Docker Compose is awesome for local development environments and cross-team collaboration by providing a unified, easily shareable configuration that ensures consistent setup across different machines, allowing developers to quickly spin up the entire microservices ecosystem with a single command, regardless of their individual development environments. For production deployments, you'd typically use a container orchestration platform like Kubernetes or Docker Swarm.

If you're looking to dive deeper into Docker and container orchestration, don't forget to check out my free Introduction to Docker eBook. It provides a solid foundation for working with Docker and will help you understand the concepts we've covered here in more depth.

Also, if you're setting up your containerized microservices and need a reliable host, consider using DigitalOcean. You can get a $200 free credit to get started!

Happy Dockerizing, and may your microservices always be scalable, resilient, and secure!