Docker, Docker Compose, and Docker Swarm

---

Docker & Docker Compose for Data Engineering

A comprehensive guide to containerizing data pipelines, databases, and analytics workflows.

Official Resources:

• Docker Documentation

• Cheatsheets:

  • Docker CLI Cheatsheet

  • Another cheatsheet

  • 3

• Dockerfile reference

• Compose File Reference

• Docker container networking, 2

• Overview of Docker functionality and best practices

---

What is Docker?

Docker packages applications with their complete runtime environments into lightweight, portable containers. Unlike virtual machines that virtualize hardware, containers virtualize at the operating system level—sharing the host kernel while maintaining strong isolation through namespaces.

Why Docker for Data Engineering?

• Reproducible environments across development, testing, and production

• Version-controlled infrastructure alongside your code

• Easy orchestration of complex data stacks (databases, ETL tools, streaming platforms)

• Simplified dependency management for Python/R/Java data tools

• Rapid environment teardown and rebuild during development

---

Core Concepts

Images vs Containers

Image: A read-only template defining what a container should be. Think of it as a class in OOP.

Container: A running instance of an image with its own filesystem, network, and process space. Think of it as an object instantiated from a class.

Key Insight: Deleting and recreating a container resets it to the original image state. Restarting a stopped container preserves changes made during its lifetime.

┌─────────────────────────────────────────────────────────────┐
│                    DOCKER ARCHITECTURE                       │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │ Docker CLI   │  │ Docker       │  │  REST API    │      │
│  │ (Commands)   │─▶│ Desktop GUI  │─▶│              │      │
│  └──────────────┘  └──────────────┘  └──────┬───────┘      │
│                                              │               │
│                                              ▼               │
│                                    ┌─────────────────┐      │
│                                    │ Docker Daemon   │      │
│                                    │  (dockerd)      │      │
│                                    └────────┬────────┘      │
│                                             │               │
│        ┌────────────────┬──────────────────┼────────┐      │
│        │                │                  │        │      │
│        ▼                ▼                  ▼        ▼      │
│  ┌──────────┐    ┌──────────┐      ┌──────────┐  ┌─────┐ │
│  │ Images   │    │Containers│      │ Volumes  │  │ Net │ │
│  └──────────┘    └──────────┘      └──────────┘  └─────┘ │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Container vs Virtual Machine

┌─────────────────────────┐  ┌─────────────────────────┐
│   VIRTUAL MACHINES      │  │      CONTAINERS         │
├─────────────────────────┤  ├─────────────────────────┤
│  ┌─────┐  ┌─────┐       │  │  ┌────┐ ┌────┐ ┌────┐  │
│  │App A│  │App B│       │  │  │App │ │App │ │App │  │
│  ├─────┤  ├─────┤       │  │  │ A  │ │ B  │ │ C  │  │
│  │Bins/│  │Bins/│       │  │  └────┘ └────┘ └────┘  │
│  │Libs │  │Libs │       │  │  ┌──────────────────┐  │
│  ├─────┤  ├─────┤       │  │  │   Docker Engine  │  │
│  │Guest│  │Guest│       │  │  └──────────────────┘  │
│  │ OS  │  │ OS  │       │  │  ┌──────────────────┐  │
│  └─────┘  └─────┘       │  │  │    Host OS       │  │
│  ┌───────────────────┐  │  │  └──────────────────┘  │
│  │   Hypervisor      │  │  │  ┌──────────────────┐  │
│  └───────────────────┘  │  │  │  Infrastructure  │  │
│  ┌───────────────────┐  │  │  └──────────────────┘  │
│  │  Infrastructure   │  │  │                         │
│  └───────────────────┘  │  │  • Lightweight          │
│                         │  │  • Fast startup         │
│  • Heavy (GBs)          │  │  • Shared OS kernel     │
│  • Slow startup         │  │  • Better resource use  │
│  • Full OS isolation    │  │                         │
└─────────────────────────┘  └─────────────────────────┘

Docker Architecture

Docker Daemon: The background service that manages containers. Must be running for Docker to function.

Docker Desktop: A GUI wrapper (optional on Linux) that manages the daemon, provides visualization, and on Mac/Windows, handles the Linux VM that actually runs Docker.

Docker CLI: Your primary interface. Communicates directly with the daemon via socket/API.

Start here: docker --help or docker <command> --help for specific guidance.

---

Essential Commands

Container Lifecycle

Run a container:

# Basic syntax
docker run [OPTIONS] IMAGE[:TAG] [COMMAND]

# Data Engineering example: Run PostgreSQL
docker run -d \
  --name postgres-de \
  -e POSTGRES_PASSWORD=mypassword \
  -e POSTGRES_DB=analytics \
  -p 5432:5432 \
  -v pgdata:/var/lib/postgresql/data \
  postgres:16

# Run Jupyter with data science stack
docker run -d \
  --name jupyter-lab \
  -p 8888:8888 \
  -v $(pwd)/notebooks:/home/jovyan/work \
  jupyter/datascience-notebook:latest

Common flags:

• -d: Detached mode (runs in background)

• -p host:container: Port mapping.

  • It's two-way communication.

    When you map ports with -p 8080:80:

    Incoming (to container):

    • Requests to localhost:8080 on your machine → forwarded to port 80 in the container

    Outgoing (from container):

    • Responses from the container's port 80 → sent back through port 8080 to your machine

    So it's a complete bidirectional channel. You send HTTP requests to port 8080, they go into the container on port 80, the application processes them, and the responses come back out the same path.

• -v volume:path: Mount volume for persistence

• -e KEY=VALUE: Set environment variables

• --name: Assign a custom name

• --network: Attach to a specific network

• --rm: Auto-remove container when stopped

• -m, --memory=""

  • limit memory allocation. Number is a positive integer. Unit can be one of b, k, m, or g

• --memory-swap=""

  • used to control the total amount of memory (RAM) plus swap space (virtual memory) that a container can use

List containers:

docker ps              # Running containers
docker ps -a           # All containers (including stopped)
docker ps -q           # Only container IDs (useful for scripts)

Stop/Start/Restart:

docker stop <container>    # Graceful shutdown (SIGTERM)
docker kill <container>    # Forceful stop (SIGKILL)
docker start <container>   # Start stopped container
docker restart <container> # Stop and start

Remove containers:

docker rm <container>      # Remove stopped container
docker rm -f <container>   # Force remove running container
docker container prune     # Remove all stopped containers

Working with Images

List images:

docker images
docker images -q  # Only IDs

Pull/Push images:

docker pull postgres:16
docker pull apache/airflow:2.7.0
docker push myusername/my-etl-pipeline:v1.0

Remove images:

docker rmi <image>
docker image prune -a  # Remove unused images

Search Docker Hub:

docker search apache-spark

---

Building Custom Images

Dockerfile Basics

A Dockerfile is Infrastructure as Code for your container image. Place it in your project root.

Data Engineering Example - Python ETL Container:

# Start from official Python image
FROM python:3.11-slim

# Set working directory
WORKDIR /app

# Install system dependencies for data tools
RUN apt-get update && apt-get install -y \
    gcc \
    g++ \
    postgresql-client \
    && rm -rf /var/lib/apt/lists/*

# Copy requirements first (layer caching optimization)
COPY requirements.txt .

# Install Python packages
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY ./src ./src
COPY ./scripts ./scripts

# Create directory for data
RUN mkdir -p /data

# Set environment variables
ENV PYTHONUNBUFFERED=1
ENV DATA_DIR=/data

# Default command
CMD ["python", "src/pipeline.py"]

Build the image:

docker build -t my-etl-pipeline:latest .
docker build -t my-etl-pipeline:v1.0 .  # With version tag

Key Dockerfile Instructions:

• FROM: Base image to build upon

• WORKDIR: Set working directory

• COPY/ADD: Copy files from host to image

• RUN: Execute commands during build (install packages, etc.)

• ENV: Set environment variables

• EXPOSE: Document which ports the container listens on

• CMD: Default command when container starts

• ENTRYPOINT: Configure container as executable

CMD vs ENTRYPOINT

Best Practices:

• Use official base images when possible

• Copy dependency files before application code (layer caching)

• Minimize layers by combining RUN commands

• Use .dockerignore to exclude unnecessary files

• Don't store secrets in images (use environment variables or secrets management)

---

Executing Commands in Containers

One-off Commands

# Execute command in running container
docker exec <container> ls /app

# Check Python version in Jupyter container
docker exec jupyter-lab python --version

# Run SQL query in PostgreSQL
docker exec postgres-de psql -U postgres -d analytics -c "SELECT COUNT(*) FROM events;"

# View running processes in container
docker exec <container> ps aux

Interactive Shell Sessions

# Start shell session (most containers use sh or bash)
docker exec -it <container> /bin/bash
docker exec -it <container> /bin/sh

# Example: Debug Python environment
docker exec -it jupyter-lab bash
> pip list
> python -c "import pandas; print(pandas.__version__)"
> exit

Flags:

• -i: Interactive mode (keep STDIN open)

• -t: Allocate pseudo-TTY (keyboard interface)

---

Monitoring & Debugging

Viewing Logs

# View all logs
docker logs <container>

# Follow logs in real-time
docker logs -f <container>

# Last N lines
docker logs --tail 50 <container>

# Logs since specific time
docker logs --since 2024-01-01T10:00:00 <container>

# Example: Monitor Airflow scheduler
docker logs -f --tail 100 airflow-scheduler

Resource Monitoring

Check resource usage:

# Live stats for all containers
docker stats

# Specific container
docker stats postgres-de

# No-stream mode (one-time snapshot)
docker stats --no-stream

View processes in container:

docker top <container>

Resource Limits

Control resource allocation to prevent containers from consuming excessive resources:

# Limit CPU and memory
docker run -d \
  --name spark-worker \
  --cpus="2.0" \
  --memory="4g" \
  --memory-swap="4g" \
  bitnami/spark:latest

# CPU shares (relative weight)
docker run -d --cpu-shares=512 postgres:16

Memory suffixes: b, k, m, g (e.g., 512m, 2g)

Resource Constraints Documentation

---

docker cp

Copy files between your host machine and a container:

# Copy a file FROM host TO container
docker cp myfile.txt my_container:/app/myfile.txt

# Copy a file FROM container TO host
docker cp my_container:/app/logs.txt ./logs.txt

# Copy an entire directory
docker cp my_container:/var/log ./container_logs

Example scenario:

# Start a container
docker run -d --name nginx_test nginx

# Copy a custom HTML file into the container
docker cp index.html nginx_test:/usr/share/nginx/html/index.html

# Copy the nginx config out to inspect it
docker cp nginx_test:/etc/nginx/nginx.conf ./nginx.conf

---

docker diff

Shows what files have been modified in a container since it started:

docker diff my_container

Example output:

C /root
A /root/.bash_history
C /var/log
A /var/log/app.log
D /tmp/oldfile.txt

• A = Added file

• C = Changed file/directory

• D = Deleted file

Example scenario:

# Run a container and make some changes
docker run -dit --name ubuntu_test ubuntu bash
docker exec ubuntu_test touch /tmp/newfile.txt
docker exec ubuntu_test rm /etc/hostname

# See what changed
docker diff ubuntu_test

Output might show:

C /tmp
A /tmp/newfile.txt
D /etc/hostname
C /root

This is useful for debugging or understanding what a container modified before committing it to an image.

---

Persistent Storage: Volumes

Containers are ephemeral by design. Volumes provide persistent storage that survives container deletion.

Volume Types

┌────────────────────────────────────────────────────────────┐
│                    DOCKER STORAGE TYPES                     │
├────────────────────────────────────────────────────────────┤
│                                                              │
│  1. NAMED VOLUMES (Recommended)                             │
│  ┌──────────────┐         ┌─────────────────────┐          │
│  │  Container   │ ───────▶│ /var/lib/postgresql │          │
│  │  (postgres)  │         │       /data         │          │
│  └──────────────┘         └─────────────────────┘          │
│         │                           │                       │
│         │                           ▼                       │
│         │                 ┌──────────────────┐             │
│         │                 │  postgres-data   │             │
│         │                 │ (Docker-managed) │             │
│         │                 └──────────────────┘             │
│                                                              │
│  2. BIND MOUNTS (Development)                               │
│  ┌──────────────┐         ┌─────────────────────┐          │
│  │  Container   │ ───────▶│    /app/code        │          │
│  │  (jupyter)   │         └─────────────────────┘          │
│  └──────────────┘                   │                       │
│                                      ▼                       │
│                          ┌──────────────────────┐           │
│                          │ /home/user/project   │           │
│                          │  (Host filesystem)   │           │
│                          └──────────────────────┘           │
│                                                              │
│  3. ANONYMOUS VOLUMES                                       │
│  ┌──────────────┐         ┌─────────────────────┐          │
│  │  Container   │ ───────▶│     /data           │          │
│  └──────────────┘         └─────────────────────┘          │
│                                    │                        │
│                                    ▼                        │
│                          ┌────────────────────┐            │
│                          │ Random hash name   │            │
│                          │ (Hard to reference)│            │
│                          └────────────────────┘            │
└────────────────────────────────────────────────────────────┘

Named Volumes (Recommended): Managed by Docker, stored in Docker's storage directory.

Anonymous Volumes: Created automatically, harder to reference later.

Bind Mounts: Direct mapping to host filesystem paths.

Working with Volumes

# Create named volume
docker volume create postgres-data
docker volume create airflow-logs

# List volumes
docker volume ls

# Inspect volume (see mount point)
docker volume inspect postgres-data

# Remove volume
docker volume rm postgres-data

# Remove unused volumes
docker volume prune

Using Volumes

Named volume example:

docker run -d \
  --name postgres-analytics \
  -v postgres-data:/var/lib/postgresql/data \
  postgres:16

Bind mount example (development):

# Mount local code directory
docker run -d \
  --name jupyter-dev \
  -v $(pwd)/notebooks:/home/jovyan/work \
  -v $(pwd)/data:/data \
  jupyter/datascience-notebook:latest

Multiple volumes:

docker run -d \
  --name airflow-worker \
  -v airflow-logs:/opt/airflow/logs \
  -v airflow-dags:/opt/airflow/dags \
  -v airflow-plugins:/opt/airflow/plugins \
  apache/airflow:2.7.0

Data Engineering Use Cases:

• Database storage (PostgreSQL, MongoDB, Elasticsearch)

• Persistent logs (Airflow, Spark)

• Shared datasets between containers

• ML model artifacts and checkpoints

• Configuration files

---

Networking

Containers need networking for inter-service communication and external access.

Network Modes

┌─────────────────────────────────────────────────────────────┐
│                    DOCKER NETWORK MODES                      │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  1. BRIDGE (Default)                                         │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  Host Machine                                        │    │
│  │  ┌────────────────────────────────────────────┐     │    │
│  │  │  Docker Bridge Network (172.17.0.0/16)     │     │    │
│  │  │                                             │     │    │
│  │  │  ┌──────────┐  ┌──────────┐  ┌──────────┐ │     │    │
│  │  │  │Container │  │Container │  │Container │ │     │    │
│  │  │  │   App    │◀▶│   DB     │◀▶│  Cache   │ │     │    │
│  │  │  │172.17.0.2│  │172.17.0.3│  │172.17.0.4│ │     │    │
│  │  │  └────┬─────┘  └──────────┘  └──────────┘ │     │    │
│  │  │       │                                     │     │    │
│  │  └───────┼─────────────────────────────────────┘     │    │
│  │          │ Port mapping (-p 8080:80)                 │    │
│  │  ┌───────▼────────┐                                  │    │
│  │  │  Host Network  │  ◀─────  Internet               │    │
│  │  │  192.168.1.10  │                                  │    │
│  │  └────────────────┘                                  │    │
│  └─────────────────────────────────────────────────────┘    │
│                                                               │
│  2. HOST (Shares host network)                               │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  Host Machine                                        │    │
│  │  ┌────────────────────────────────────────────┐     │    │
│  │  │  Host Network Interface                     │     │    │
│  │  │                                             │     │    │
│  │  │  ┌──────────┐                               │     │    │
│  │  │  │Container │ (No isolation)                │     │    │
│  │  │  │ --net=host│                              │     │    │
│  │  │  └──────────┘                               │     │    │
│  │  └─────────────────────────────────────────────┘     │    │
│  └─────────────────────────────────────────────────────┘    │
│                                                               │
│  3. NONE (Isolated)                                          │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  Host Machine                                        │    │
│  │  ┌──────────┐                                        │    │
│  │  │Container │  No network access                     │    │
│  │  │--net=none│  (Security/isolation)                  │    │
│  │  └──────────┘                                        │    │
│  └─────────────────────────────────────────────────────┘    │
│                                                               │
│  4. CUSTOM BRIDGE (User-defined)                             │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  Host Machine                                        │    │
│  │  ┌────────────────────────────────────────────┐     │    │
│  │  │  Custom Network: data-pipeline              │     │    │
│  │  │                                             │     │    │
│  │  │  ┌──────────┐         ┌──────────┐         │     │    │
│  │  │  │   ETL    │◀───────▶│ Postgres │         │     │    │
│  │  │  │  Job     │  DNS:   │          │         │     │    │
│  │  │  │          │postgres │          │         │     │    │
│  │  │  └──────────┘         └──────────┘         │     │    │
│  │  │  (Automatic DNS resolution by name)        │     │    │
│  │  └─────────────────────────────────────────────┘     │    │
│  └─────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘

Bridge (default): Private network for container communication.

Host: Container shares host's network stack.

None: No networking (isolated mode).

Custom networks: User-defined bridges with DNS resolution.

Working with Networks

# Create custom network
docker network create data-pipeline-net

# List networks
docker network ls

# Inspect network
docker network inspect data-pipeline-net

# Remove network
docker network rm data-pipeline-net

Connecting Containers

Example: Connect application to database

# Create network
docker network create analytics-net

# Run PostgreSQL on network
docker run -d \
  --name postgres \
  --network analytics-net \
  -e POSTGRES_PASSWORD=secret \
  postgres:16

# Run Python app that connects to database
docker run -d \
  --name etl-job \
  --network analytics-net \
  -e DB_HOST=postgres \
  -e DB_PORT=5432 \
  -e DB_NAME=analytics \
  my-etl-pipeline:latest

Network Communication Diagram:

┌──────────────────────────────────────────────────────────┐
│            Custom Network: analytics-net                 │
│                                                           │
│   ┌─────────────────┐           ┌──────────────────┐    │
│   │   ETL Job       │           │   PostgreSQL     │    │
│   │  Container      │           │   Container      │    │
│   │                 │           │                  │    │
│   │  DB_HOST=       │──────────▶│  Listening on    │    │
│   │  "postgres"     │  Connects │  port 5432       │    │
│   │                 │  via DNS  │                  │    │
│   │  Python app     │◀──────────│  Returns data    │    │
│   │  makes query    │           │                  │    │
│   └─────────────────┘           └──────────────────┘    │
│                                                           │
│   Docker provides automatic DNS resolution:              │
│   "postgres" → 172.18.0.2 (internal IP)                 │
└──────────────────────────────────────────────────────────┘

Key Feature: Docker provides automatic DNS resolution. Container etl-job can connect to postgres by name, not IP address.

Network Isolation

Offline mode (for security or testing):

docker run -d --network none my-untrusted-container:latest

Test connectivity:

docker exec <container> ping google.com -W 2
docker exec <container> curl http://api.example.com

---

Here is your modernized, up-to-date cheatsheet.

Key Changes Applied:

1. Commands: Updated from docker-compose (v1) to docker compose (v2).

2. File Name: Updated to compose.yaml, which is the preferred standard for V2 (though docker-compose.yml still works).

3. Version Key: Removed the version: '3.8' line. The new Compose Specification (V2) defaults to the latest schema automatically, making the version line obsolete.

---

Docker Compose (V2) Cheatsheet

Docker Compose orchestrates multi-container applications using a declarative YAML file. Essential for data engineering where you typically run databases, message queues, ETL tools, and monitoring together.

Why Use Compose?

┌─────────────────────────────────────────────────────────────┐
│              WITHOUT COMPOSE vs WITH COMPOSE                │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  WITHOUT COMPOSE (Manual Setup)                             │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  $ docker network create data-net                     │  │
│  │  $ docker volume create postgres-data                 │  │
│  │  $ docker run -d --name postgres --network data-net \ │  │
│  │      -e POSTGRES_PASSWORD=pass \                      │  │
│  │      -v postgres-data:/var/lib/postgresql/data \      │  │
│  │      -p 5432:5432 postgres:16                         │  │
│  │  $ docker run -d --name redis --network data-net ...  │  │
│  │  $ docker run -d --name app --network data-net ...    │  │
│  │  ... and so on for each service                       │  │
│  └───────────────────────────────────────────────────────┘  │
│                                                             │
│  WITH COMPOSE (Single Command)                              │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  $ docker compose up -d                               │  │
│  │                                                       │  │
│  │  ✓ Creates network automatically                      │  │
│  │  ✓ Creates volumes automatically                      │  │
│  │  ✓ Starts all services in correct order               │  │
│  │  ✓ Configures dependencies                            │  │
│  │  ✓ Sets up DNS resolution                             │  │
│  └───────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘

• Define entire data stack in one file

• Single command to start/stop all services

• Automatic network creation and DNS resolution (service names = hostnames)

• Service dependencies and health checks

Basic Compose File

File: compose.yaml (Preferred) or docker-compose.yml

services:
  postgres:
    image: postgres:16
    container_name: postgres-de
    environment:
      POSTGRES_USER: dataeng
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: warehouse
    volumes:
      - postgres-data:/var/lib/postgresql/data
      - ./init-scripts:/docker-entrypoint-initdb.d
    ports:
      - "5432:5432"
    networks:
      - data-net
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U dataeng"]
      interval: 10s
      timeout: 5s
      retries: 5

  pgadmin:
    image: dpage/pgadmin4:latest
    container_name: pgadmin
    environment:
      PGADMIN_DEFAULT_EMAIL: [e-mail]
      PGADMIN_DEFAULT_PASSWORD: admin
    ports:
      - "5050:80"
    networks:
      - data-net
    depends_on:
      postgres:
        condition: service_healthy

volumes:
  postgres-data:

networks:
  data-net:
    driver: bridge

Essential Compose Commands

# Start all services (detached mode)
docker compose up -d

# View running services (Project specific)
docker compose ps

# List all running compose projects globally
docker compose ls

# View logs
docker compose logs
docker compose logs -f postgres      # Follow specific service
docker compose logs --tail=50 -f     # Last 50 lines, follow

# Stop services (keeps containers/state)
docker compose stop

# Stop and remove containers and networks
docker compose down

# Stop, remove containers, networks AND volumes (Reset state)
docker compose down -v

# Restart specific service
docker compose restart postgres

# Execute command in running service
docker compose exec postgres psql -U dataeng -d warehouse

# Build/rebuild services
docker compose build
docker compose up -d --build

# Scale services (e.g., workers)
docker compose up -d --scale spark-worker=3

more commands

Common Docker Compose Commands

Here are some of the most frequently used commands and their functions:

Command	Description
docker compose up	Creates and starts all services defined in the Compose file. Use with the -d flag for detached mode (running in the background).
docker compose down	Stops and removes containers, networks, and volumes created by up.
docker compose ps	Lists the running containers for the current project.
docker compose logs	Views the aggregated log output from all containers.
docker compose build	Builds or rebuilds service images.
docker compose pull	Pulls service images from a registry (e.g., Docker Hub).
docker compose start	Starts existing, stopped service containers.
docker compose stop	Stops running service containers without removing them.
docker compose restart	Restarts service containers.
docker compose exec	Executes a command in a running container.
docker compose run	Runs a one-off command on a service (e.g., running tests or migrations).

For a full list of commands and options, refer to the Docker Compose CLI reference.

Data Engineering Stack Example

Complete pipeline with database, message queue, and processing.

services:
  # PostgreSQL Data Warehouse
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: ${POSTGRES_USER:-dataeng}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-changeme}
      POSTGRES_DB: ${POSTGRES_DB:-warehouse}
    volumes:
      - postgres-data:/var/lib/postgresql/data
      - ./sql/init:/docker-entrypoint-initdb.d
    ports:
      - "${POSTGRES_PORT:-5432}:5432"
    networks:
      - data-pipeline
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-dataeng}"]
      interval: 10s
      timeout: 5s
      retries: 5

  # Redis for caching and temporary storage
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    networks:
      - data-pipeline
    volumes:
      - redis-data:/data

  # Apache Kafka (with Zookeeper)
  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    networks:
      - data-pipeline
    volumes:
      - kafka-data:/var/lib/kafka/data

  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    networks:
      - data-pipeline
    volumes:
      - zookeeper-data:/var/lib/zookeeper

  # ETL Worker (Python App)
  etl-worker:
    build:
      context: ./etl
      dockerfile: Dockerfile
    environment:
      DB_HOST: postgres
      DB_PORT: 5432
      DB_NAME: ${POSTGRES_DB:-warehouse}
      DB_USER: ${POSTGRES_USER:-dataeng}
      DB_PASSWORD: ${POSTGRES_PASSWORD:-changeme}
      REDIS_HOST: redis
      KAFKA_BOOTSTRAP_SERVERS: kafka:29092
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_started
      kafka:
        condition: service_started
    networks:
      - data-pipeline
    volumes:
      - ./etl/src:/app/src
      - ./data:/data
      - ./logs:/logs

  # Jupyter for analysis
  jupyter:
    image: jupyter/datascience-notebook:latest
    ports:
      - "8888:8888"
    environment:
      JUPYTER_ENABLE_LAB: "yes"
    volumes:
      - ./notebooks:/home/jovyan/work
      - ./data:/home/jovyan/data
    networks:
      - data-pipeline

volumes:
  postgres-data:
  redis-data:
  kafka-data:
  zookeeper-data:

networks:
  data-pipeline:
    driver: bridge

Environment Variables

1. Using .env file (Default)

Create .env in the same directory as compose.yaml. Compose automatically loads these.

POSTGRES_USER=dataeng
POSTGRES_PASSWORD=supersecret
POSTGRES_DB=warehouse
POSTGRES_PORT=5432

Reference in YAML: ${VAR_NAME} or ${VAR_NAME:-default_value}.

2. Override for different environments

docker compose --env-file .env.production up -d

Advanced Compose Features

Service Dependencies with Health Checks

Ensures services don't start until their dependencies are actually ready (not just running).

services:
  app:
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_started

Resource Limits

Limit CPU and RAM to prevent crashes on your local machine.

services:
  spark-worker:
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 4G
        reservations:
          cpus: '1.0'
          memory: 2G

Profiles (Selective Startup)

Great for running only parts of the stack (e.g., only dev tools, or skipping heavy monitoring).

services:
  postgres:
    # No profile: Always runs
  
  jupyter:
    profiles: ["dev"]  # Only runs if 'dev' profile is active
  
  monitoring:
    profiles: ["monitoring"]

Command: docker compose --profile dev up -d

---

Authentication & Security

Docker Hub Authentication

Login to Docker Hub:

docker login

# With username
docker login -u myusername

# Logout
docker logout

Tag and push images:

# Tag with username/repository
docker tag my-etl-pipeline:latest myusername/my-etl-pipeline:latest
docker tag my-etl-pipeline:latest myusername/my-etl-pipeline:v1.0

# Push to Docker Hub
docker push myusername/my-etl-pipeline:latest
docker push myusername/my-etl-pipeline:v1.0

Alternative Container Registries

Amazon ECR (Elastic Container Registry):

# Authenticate
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin 123456789.dkr.ecr.us-east-1.amazonaws.com

# Tag and push
docker tag my-image:latest 123456789.dkr.ecr.us-east-1.amazonaws.com/my-repo:latest
docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/my-repo:latest

Google Artifact Registry:

# Authenticate
gcloud auth configure-docker us-central1-docker.pkg.dev

# Tag and push
docker tag my-image:latest us-central1-docker.pkg.dev/project-id/repo-name/my-image:latest
docker push us-central1-docker.pkg.dev/project-id/repo-name/my-image:latest

GitHub Container Registry (GHCR):

# Create Personal Access Token with write:packages scope
echo $GITHUB_TOKEN | docker login ghcr.io -u USERNAME --password-stdin

# Tag and push
docker tag my-image:latest ghcr.io/username/my-image:latest
docker push ghcr.io/username/my-image:latest

Azure Container Registry:

# Login
az acr login --name myregistry

# Tag and push
docker tag my-image:latest myregistry.azurecr.io/my-image:latest
docker push myregistry.azurecr.io/my-image:latest

Security Best Practices

1. Use secrets, not environment variables for sensitive data:

Docker Compose with secrets (Swarm mode):

services:
  postgres:
    secrets:
      - postgres_password
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password

secrets:
  postgres_password:
    file: ./secrets/postgres_password.txt

2. Run containers as non-root:

FROM python:3.11-slim

RUN useradd -m -u 1000 dataeng
USER dataeng

WORKDIR /app

3. Use official images from trusted sources

4. Scan images for vulnerabilities:

docker scan my-etl-pipeline:latest

5. Don't store secrets in images:

• Use environment variables

• Use Docker secrets

• Use external secret management (Vault, AWS Secrets Manager)

6. Use .dockerignore:

.git
.env
*.log
__pycache__
*.pyc
.pytest_cache
node_modules
.venv
credentials.json

---

Data Engineering Workflow Examples

Example 1: PostgreSQL + dbt + Jupyter

version: '3.8'

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: dbt_user
      POSTGRES_PASSWORD: dbt_pass
      POSTGRES_DB: analytics
    volumes:
      - postgres-data:/var/lib/postgresql/data
    ports:
      - "5432:5432"

  dbt:
    build:
      context: ./dbt
    environment:
      DBT_PROFILES_DIR: /root/.dbt
    volumes:
      - ./dbt:/usr/app
      - ~/.dbt:/root/.dbt
    depends_on:
      - postgres
    command: dbt run

  jupyter:
    image: jupyter/datascience-notebook:latest
    ports:
      - "8888:8888"
    volumes:
      - ./notebooks:/home/jovyan/work
    depends_on:
      - postgres

volumes:
  postgres-data:

Example 2: Airflow Data Pipeline

version: '3.8'

x-airflow-common:
  &airflow-common
  image: apache/airflow:2.7.0
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: LocalExecutor
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
  user: "${AIRFLOW_UID:-50000}:0"
  depends_on:
    &airflow-common-depends-on
    postgres:
      condition: service_healthy

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "airflow"]
      interval: 5s
      retries: 5

  airflow-webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 10s
      timeout: 10s
      retries: 5

  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
    healthcheck:
      test: ["CMD-SHELL", 'airflow jobs check --job-type SchedulerJob']
      interval: 10s
      timeout: 10s
      retries: 5

  airflow-init:
    <<: *airflow-common
    entrypoint: /bin/bash
    command:
      - -c
      - |
        airflow db init
        airflow users create \
          --username admin \
          --firstname admin \
          --lastname admin \
          --role Admin \
          --email [e-mail] \
          --password admin
    environment:
      <<: *airflow-common-env
      _AIRFLOW_DB_UPGRADE: 'true'
      _AIRFLOW_WWW_USER_CREATE: 'true'

volumes:
  postgres-data:

Example 3: Spark Cluster

version: '3.8'

services:
  spark-master:
    image: bitnami/spark:latest
    environment:
      - SPARK_MODE=master
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
    ports:
      - "8080:8080"
      - "7077:7077"
    volumes:
      - ./spark-apps:/opt/spark-apps
      - ./spark-data:/opt/spark-data

  spark-worker:
    image: bitnami/spark:latest
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark-master:7077
      - SPARK_WORKER_MEMORY=2G
      - SPARK_WORKER_CORES=2
    depends_on:
      - spark-master
    volumes:
      - ./spark-apps:/opt/spark-apps
      - ./spark-data:/opt/spark-data

---

CI/CD Integration

Modern deployment workflow with Docker:

┌──────────────────────────────────────────────────────────────┐
│              DOCKER CI/CD PIPELINE                            │
├──────────────────────────────────────────────────────────────┤
│                                                                │
│  1. DEVELOPMENT                                               │
│  ┌──────────────┐                                             │
│  │  Developer   │  Writes code, commits to Git               │
│  │  (You)       │  Pushes branch to GitHub                   │
│  └──────┬───────┘                                             │
│         │                                                      │
│         ▼                                                      │
│  2. SOURCE CONTROL                                            │
│  ┌──────────────┐                                             │
│  │   GitHub     │  Pull Request created                      │
│  │   GitLab     │  Code review & approval                    │
│  └──────┬───────┘                                             │
│         │ Merge to main                                       │
│         ▼                                                      │
│  3. CI/CD TRIGGER                                             │
│  ┌──────────────┐                                             │
│  │ GitHub       │  Webhook triggers pipeline                 │
│  │ Actions      │  Automated workflow starts                 │
│  └──────┬───────┘                                             │
│         │                                                      │
│         ▼                                                      │
│  4. BUILD & TEST                                              │
│  ┌────────────────────────────────────────┐                  │
│  │  • Build Docker image                  │                  │
│  │  • Run tests in container              │                  │
│  │  • Security scan image                 │                  │
│  │  • Tag with version/commit SHA         │                  │
│  └──────┬─────────────────────────────────┘                  │
│         │                                                      │
│         ▼                                                      │
│  5. REGISTRY                                                  │
│  ┌──────────────┐                                             │
│  │ Docker Hub   │  Push tagged image                         │
│  │ ECR/GHCR     │  myapp:v1.2.3, myapp:latest                │
│  └──────┬───────┘                                             │
│         │                                                      │
│         ▼                                                      │
│  6. DEPLOYMENT                                                │
│  ┌────────────────────────────────────────┐                  │
│  │  Kubernetes / ECS / Docker Swarm       │                  │
│  │  • Pulls latest image                  │                  │
│  │  • Rolling update (zero downtime)      │                  │
│  │  • Health checks                       │                  │
│  │  • Old containers replaced with new    │                  │
│  └────────────────────────────────────────┘                  │
│                                                                │
│  7. PRODUCTION                                                │
│  ┌──────────────┐                                             │
│  │  Running     │  New version serving traffic               │
│  │  Containers  │  Monitoring & logging active               │
│  └──────────────┘                                             │
└──────────────────────────────────────────────────────────────┘

1. Developer pushes code to Git repository

2. CI/CD pipeline triggers (GitHub Actions, GitLab CI, Jenkins)

3. Pipeline builds Docker image

4. Pipeline runs tests in container

5. Pipeline tags image with version/commit SHA

6. Pipeline pushes image to registry (Docker Hub, ECR, GHCR)

7. Deployment system pulls latest image

8. Old containers are replaced with new ones

Example GitHub Action:

name: Build and Push Docker Image

on:
  push:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Login to GitHub Container Registry
        uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      
      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          tags: |
            ghcr.io/${{ github.repository }}:latest
            ghcr.io/${{ github.repository }}:${{ github.sha }}

---

Docker Swarm

Docs

Docker Swarm is Docker's native clustering and orchestration solution. It turns multiple Docker hosts into a single virtual host, enabling container deployment across a cluster of machines with built-in load balancing, service discovery, and high availability.

Swarm vs Compose vs Kubernetes

┌─────────────────────────────────────────────────────────────┐
│              ORCHESTRATION COMPARISON                        │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  DOCKER COMPOSE                                              │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  Single Host                                         │    │
│  │  ┌──────┐  ┌──────┐  ┌──────┐                       │    │
│  │  │ App  │  │  DB  │  │Cache │                       │    │
│  │  └──────┘  └──────┘  └──────┘                       │    │
│  │  • Development/Testing                               │    │
│  │  • Single machine                                    │    │
│  │  • No high availability                              │    │
│  └─────────────────────────────────────────────────────┘    │
│                                                               │
│  DOCKER SWARM                                                │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  Swarm Cluster                                       │    │
│  │  ┌─────────────┐  ┌─────────────┐  ┌──────────────┐│    │
│  │  │  Manager    │  │  Worker 1   │  │   Worker 2   ││    │
│  │  │  Node       │  │  ┌──────┐   │  │   ┌──────┐   ││    │
│  │  │  Orchestrate│  │  │ App  │   │  │   │ App  │   ││    │
│  │  │  & Schedule │  │  └──────┘   │  │   └──────┘   ││    │
│  │  └─────────────┘  └─────────────┘  └──────────────┘│    │
│  │  • Production (small/medium scale)                  │    │
│  │  • Simple setup                                     │    │
│  │  • Built into Docker                                │    │
│  │  • Native Docker API                                │    │
│  └─────────────────────────────────────────────────────┘    │
│                                                               │
│  KUBERNETES                                                  │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  K8s Cluster                                         │    │
│  │  ┌──────────┐  ┌─────────┐  ┌─────────┐ ┌────────┐ │    │
│  │  │  Control │  │  Node 1 │  │  Node 2 │ │ Node 3 │ │    │
│  │  │  Plane   │  │  ┌───┐  │  │  ┌───┐  │ │ ┌───┐  │ │    │
│  │  │  ┌────┐  │  │  │Pod│  │  │  │Pod│  │ │ │Pod│  │ │    │
│  │  │  │API │  │  │  └───┘  │  │  └───┘  │ │ └───┘  │ │    │
│  │  │  └────┘  │  └─────────┘  └─────────┘ └────────┘ │    │
│  │  • Production (large scale)                         │    │
│  │  • Complex features                                 │    │
│  │  • Industry standard                                │    │
│  │  • Steeper learning curve                           │    │
│  └─────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘

When to Use Docker Swarm

Good for:

• Small to medium production deployments

• Teams already familiar with Docker

• Simple orchestration needs

• When you want native Docker CLI integration

• Quick setup without complex configuration

Not ideal for:

• Very large scale (1000+ containers)

• Complex networking requirements

• Advanced scheduling needs

• When industry-standard tooling is required (Kubernetes dominates)

Swarm Architecture

┌──────────────────────────────────────────────────────────────┐
│                  DOCKER SWARM CLUSTER                         │
├──────────────────────────────────────────────────────────────┤
│                                                                │
│  ┌────────────────────────────────────────────────────────┐  │
│  │              MANAGER NODES (Raft Consensus)            │  │
│  │  ┌───────────────┐  ┌───────────────┐  ┌────────────┐ │  │
│  │  │   Manager 1   │  │   Manager 2   │  │ Manager 3  │ │  │
│  │  │   (Leader)    │◀▶│               │◀▶│            │ │  │
│  │  │               │  │               │  │            │ │  │
│  │  │ • Orchestrate │  │ • Backup      │  │ • Backup   │ │  │
│  │  │ • Schedule    │  │ • Standby     │  │ • Standby  │ │  │
│  │  │ • Maintain    │  │               │  │            │ │  │
│  │  │   State       │  │               │  │            │ │  │
│  │  └───────┬───────┘  └───────┬───────┘  └─────┬──────┘ │  │
│  └──────────┼──────────────────┼─────────────────┼────────┘  │
│             │                  │                 │            │
│  ┌──────────┼──────────────────┼─────────────────┼────────┐  │
│  │          │    WORKER NODES  │                 │        │  │
│  │  ┌───────▼───────┐  ┌───────▼───────┐  ┌─────▼──────┐ │  │
│  │  │   Worker 1    │  │   Worker 2    │  │  Worker 3  │ │  │
│  │  │               │  │               │  │            │ │  │
│  │  │  ┌─────────┐  │  │  ┌─────────┐  │  │ ┌────────┐ │ │  │
│  │  │  │Container│  │  │  │Container│  │  │ │Contain.│ │ │  │
│  │  │  │ (Task)  │  │  │  │ (Task)  │  │  │ │(Task)  │ │ │  │
│  │  │  └─────────┘  │  │  └─────────┘  │  │ └────────┘ │ │  │
│  │  │  ┌─────────┐  │  │  ┌─────────┐  │  │            │ │  │
│  │  │  │Container│  │  │  │Container│  │  │            │ │  │
│  │  │  └─────────┘  │  │  └─────────┘  │  │            │ │  │
│  │  └───────────────┘  └───────────────┘  └────────────┘ │  │
│  └──────────────────────────────────────────────────────────┘  │
│                                                                │
│  Service Discovery & Load Balancing:                          │
│  External Request → Ingress Network → Any Node → Routing Mesh │
│                  → Correct Container (regardless of location)  │
└──────────────────────────────────────────────────────────────┘

Key Swarm Concepts

Nodes: Individual Docker hosts in the cluster

• Manager nodes: Control the cluster, schedule tasks, maintain state

• Worker nodes: Execute containers (tasks)

Services: The definition of tasks to run (like Compose services)

• Replicated services: Run specified number of replicas across the cluster

• Global services: Run one task on every node

Tasks: Individual container instances running as part of a service

Stack: Group of services deployed together (like docker-compose.yml for Swarm)

Basic Swarm Commands

Initialize a Swarm:

# On manager node
docker swarm init --advertise-addr <MANAGER-IP>

# Output provides join token for workers
# To get join token again:
docker swarm join-token worker
docker swarm join-token manager

Join Swarm:

# On worker nodes
docker swarm join --token <TOKEN> <MANAGER-IP>:2377

Deploy a stack:

# Using docker-compose.yml-like file
docker stack deploy -c docker-compose.yml data-pipeline

# List stacks
docker stack ls

# List services in stack
docker stack services data-pipeline

# Remove stack
docker stack rm data-pipeline

Manage services:

# Create service
docker service create --name web --replicas 3 -p 8080:80 nginx

# List services
docker service ls

# Scale service
docker service scale web=5

# Inspect service
docker service inspect web

# View service logs
docker service logs web

# Update service (rolling update)
docker service update --image nginx:1.21 web

# Remove service
docker service rm web

Node management:

# List nodes
docker node ls

# Inspect node
docker node inspect <NODE-ID>

# Promote worker to manager
docker node promote <NODE-ID>

# Drain node (stop scheduling new tasks)
docker node update --availability drain <NODE-ID>

# Remove node from swarm (on the node itself)
docker swarm leave

Swarm Compose File Example

docker-compose.yml for Swarm:

version: '3.8'

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: secret
    volumes:
      - postgres-data:/var/lib/postgresql/data
    networks:
      - data-net
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager  # Only on manager nodes
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3

  etl-worker:
    image: myusername/etl-pipeline:latest
    environment:
      DB_HOST: postgres
      DB_PORT: 5432
    networks:
      - data-net
    deploy:
      replicas: 3  # 3 instances across cluster
      update_config:
        parallelism: 1  # Update 1 at a time
        delay: 10s
      restart_policy:
        condition: on-failure
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M

  visualizer:
    image: dockersamples/visualizer
    ports:
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - data-net
    deploy:
      placement:
        constraints:
          - node.role == manager

volumes:
  postgres-data:

networks:
  data-net:
    driver: overlay  # Overlay network for multi-host communication

Key Swarm-specific features:

• deploy section defines deployment strategy

• replicas sets number of container instances

• placement.constraints controls node placement

• update_config defines rolling update strategy

• resources sets CPU/memory limits

• overlay network enables cross-host communication

Deploy the stack:

docker stack deploy -c docker-compose.yml data-pipeline

Swarm Secrets Management

Swarm has built-in secrets management (unlike standalone Docker):

# Create secret from file
echo "supersecretpassword" | docker secret create postgres_password -

# Create secret from stdin
docker secret create db_connection - < connection_string.txt

# List secrets
docker secret ls

# Use secret in service
docker service create \
  --name postgres \
  --secret postgres_password \
  -e POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password \
  postgres:16

In docker-compose.yml:

version: '3.8'

services:
  postgres:
    image: postgres:16
    secrets:
      - postgres_password
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
    deploy:
      replicas: 1

secrets:
  postgres_password:
    external: true  # Already created with docker secret create

Swarm Networking

Overlay networks enable container communication across hosts:

# Create overlay network
docker network create --driver overlay --attachable data-overlay

# Deploy services on overlay network
docker service create --network data-overlay --name web nginx
docker service create --network data-overlay --name api myapi:latest

Ingress networking: Built-in load balancing

• Any node in the cluster can receive traffic on published ports

• Requests are automatically routed to healthy containers

• Works even if the node doesn't run the service

Monitoring Swarm

# View service status
docker service ps <service-name>

# Check node status
docker node ls

# View service logs
docker service logs -f <service-name>

# Inspect service
docker service inspect <service-name>

When to Choose What

Use Docker Compose when:

• Local development

• Single-host deployments

• Simple testing environments

Use Docker Swarm when:

• Small to medium production deployments (< 100 nodes)

• Need high availability without complexity

• Team is comfortable with Docker but not Kubernetes

• Want built-in secrets management

• Need quick setup

Use Kubernetes when:

• Large-scale production (100+ nodes)

• Complex microservices architecture

• Need advanced features (operators, custom resources)

• Industry-standard tooling required

• Have ops team with K8s expertise

---

Troubleshooting

Container won't start:

docker logs <container>
docker inspect <container>

Port already in use:

# Find process using port
lsof -i :5432  # macOS/Linux
netstat -ano | findstr :5432  # Windows

# Change port mapping
docker run -p 5433:5432 postgres:16

Permission denied (volumes):

# Check file ownership
ls -la /path/to/volume

# Run container with specific user
docker run --user $(id -u):$(id -g) ...

Out of disk space:

# Check Docker disk usage
docker system df

# Clean up
docker system prune -a --volumes

Container networking issues:

# Check container IP
docker inspect <container> | grep IPAddress

# Test connectivity
docker exec <container> ping <other-container-name>

# Check DNS resolution
docker exec <container> nslookup postgres

---

Quick Reference

Most Used Commands:

docker ps                                    # List running containers
docker-compose up -d                         # Start services
docker-compose down                          # Stop services
docker logs -f <container>                   # View logs
docker exec -it <container> /bin/bash        # Shell into container
docker-compose exec <service> <command>      # Run command in service
docker volume ls                             # List volumes
docker network ls                            # List networks
docker system prune -a                       # Clean up everything

Data Engineering Essentials:

# PostgreSQL
docker run -d -p 5432:5432 -v pgdata:/var/lib/postgresql/data \
  -e POSTGRES_PASSWORD=pass postgres:16

# MongoDB
docker run -d -p 27017:27017 -v mongodata:/data/db mongo:7

# Redis
docker run -d -p 6379:6379 -v redisdata:/data redis:7-alpine

# Elasticsearch
docker run -d -p 9200:9200 -e "discovery.type=single-node" \
  -v esdata:/usr/share/elasticsearch/data \
  elasticsearch:8.11.0

# Apache Kafka (requires compose - see examples above)

---

Additional Resources

Official Documentation:

• Dockerfile Best Practices

• Compose Specification

• Docker Security

Data Engineering Tools with Docker:

• Apache Airflow Docker

• dbt with Docker

• Kafka Docker

• Spark Docker

Container Registries:

• Docker Hub

• Amazon ECR

• Google Artifact Registry

• GitHub Container Registry

• Harbor (self-hosted)