Docker BuildKit

BuildKit is the next-generation build engine for Docker. It was introduced in Docker 18.09 and is now the default engine.

Key Advantages over the Legacy Builder

• Concurrent Execution: BuildKit can analyze the dependency graph of your Dockerfile and run independent stages in parallel.
• Improved Caching: Supports more efficient caching, including remote cache backends.
• Secret Mounting: Allows you to pass secrets (like API keys or SSH keys) to the build process without them ever being stored in the final image or build metadata.
• SSH Forwarding: Can forward your host’s SSH agent to the container during build time to access private repositories.
• Better Output: Provides a much more readable, interactive progress display.

New Dockerfile Instructions (--mount)

BuildKit introduces the --mount flag for RUN instructions:

• RUN --mount=type=cache: Persistent cache for package managers (e.g., npm or apt cache).
• RUN --mount=type=secret: Access a secret file during the build.
• RUN --mount=type=ssh: Use SSH keys from the host.

Enabling BuildKit

If not already the default, it can be enabled via the environment variable: export DOCKER_BUILDKIT=1

Docker Compose

Docker Compose is a tool for defining and running multi-container applications. It uses a YAML file (docker-compose.yml) to configure your application’s services, networks, and volumes.

Key Features

• Single Command Orchestration: With a single command (docker compose up), you can create and start all the services defined in your configuration.
• Service Isolation: Each service runs in its own container but can communicate with others over a common network.
• Environment Consistency: Ensures that the entire stack (web, DB, cache) is set up identically across different environments (dev, staging, etc.).
• Project Scoping: Compose uses a project name (usually the directory name) to isolate environments on the same host.

Core Components of docker-compose.yml

1. services: Defines the individual containers (e.g., web, db).
2. networks: Defines the virtual networks the containers connect to.
3. volumes: Defines the persistent storage shared or used by services.
4. build: Instructions for building an image from a Dockerfile.
5. image: Specifies a pre-built image to use.

Docker Health Checks

A container’s status (e.g., Up 5 minutes) only tells you if the process is running, not if the application is actually healthy and responding to requests.

The HEALTHCHECK Instruction

The HEALTHCHECK instruction tells Docker how to test a container to check that it is still working.

Parameters

• --interval=DURATION (default: 30s): How often to run the check.
• --timeout=DURATION (default: 30s): How long to wait for the check to complete.
• --start-period=DURATION (default: 0s): How long to wait for the app to start up before beginning checks.
• --retries=N (default: 3): How many consecutive failures are needed to mark the container as unhealthy.

Health Statuses

1. starting: The container is starting and the start-period hasn’t elapsed, or the first check hasn’t run.
2. healthy: The last health check was successful.
3. unhealthy: The health check failed N times in a row.

Why Use It?

Orchestrators like Docker Compose and Kubernetes use health status to decide when to restart a container or when to start routing traffic to it. Without a health check, a “zombie” container (process running but app deadlocked or crashed internally) will stay in the rotation.

Docker Image Optimization

Building efficient Docker images involves two main goals: reducing the final image size and minimizing the build time through efficient layer caching.

1. Layer Caching

Docker processes each instruction in a Dockerfile and creates a new layer. These layers are cached.

• Invalidation: If an instruction changes (or a file it copies changes), that layer and all subsequent layers are invalidated and must be rebuilt.
• Optimization: Order your instructions from “least likely to change” to “most likely to change.” For example, install dependencies before copying your source code.

2. The .dockerignore File

Similar to .gitignore, a .dockerignore file tells Docker which files or directories to exclude from the build context.

• Why it matters: It prevents large or sensitive files (like node_modules, .git, .env, or build binaries) from being sent to the Docker daemon.
• Benefits: Faster builds (smaller context) and smaller images (if using COPY . .).

3. Minimizing Layers

While modern Docker versions are better at handling many layers, combining related commands into a single RUN instruction (using && and ``) is still a common practice to reduce metadata overhead and ensure temporary files are deleted in the same layer they were created.

Docker Image Layers

A Docker image consists of a series of read-only layers, each representing an instruction in the image’s Dockerfile. These layers are stacked on top of each other to form the final image.

How Layers Work

• Union File System (UnionFS): Docker uses UnionFS to combine these layers into a single coherent file system.
• Immutability: Each layer is immutable. Once created, it never changes.
• Caching: Docker caches layers. If a Dockerfile instruction hasn’t changed, Docker reuses the existing layer from the cache during the build process, significantly speeding up builds.
• Copy-on-Write (CoW): When a container is started from an image, Docker adds a thin writable layer (the “container layer”) on top of the stack. Any changes made to the running container (like writing new files or modifying existing ones) are written to this thin layer.

Optimization Strategy

To keep images small and builds fast:

1. Order matters: Place instructions that change frequently (like COPY . .) as late as possible in the Dockerfile.
2. Chain commands: Combine multiple RUN commands using && to reduce the number of layers.
3. Clean up: Remove temporary files and package manager caches in the same RUN instruction where they were created.

Minimal Base Images

For many production applications, especially those compiled into static binaries (like Go, Rust, or C++), you don’t need a full Linux distribution (like Ubuntu or even Alpine) inside your container.

1. The scratch Image

• What it is: A completely empty, zero-byte image. It is the most minimal base possible in Docker.
• Use Case: Perfect for statically linked binaries. Your container will contain only your executable.
• Pros: Smallest possible size, zero attack surface (no shell, no ls, no libc).
• Cons: Extremely hard to debug (you can’t exec into it). You must handle all dependencies (like CA certificates) yourself.

2. Distroless Images (by Google)

• What it is: “Distroless” images contain only your application and its runtime dependencies. They do not contain package managers, shells, or any other programs you would expect to find in a standard Linux distribution.
• Use Case: Ideal for runtimes like Node.js, Python, or Java where you need some shared libraries but don’t want the security risk of a full shell.
• Pros: Much more secure than standard bases, easier than scratch because they include common requirements like glibc and CA certs.
• Cons: Still difficult to debug; requires a multi-stage build.

Comparison

Feature	Standard (Ubuntu/Alpine)	Distroless	Scratch
Size	Large (Ubuntu) / Small (Alpine)	Very Small	Zero
Shell	Yes	No	No
Package Manager	Yes	No	No
Security	Lower	High	Maximum

Multi-stage Builds

Multi-stage builds allow you to use multiple FROM statements in your Dockerfile. Each FROM instruction begins a new stage of the build. You can selectively copy artifacts from one stage to another, leaving behind everything you don’t want in the final image.

Key Benefits

• Smaller Images: By excluding build-time dependencies (compilers, SDKs, build tools) from the final runtime image, you significantly reduce the attack surface and download size.
• Simplified CI/CD: You don’t need to maintain separate Dockerfiles for building and running. Everything is contained in one file.
• Layer Optimization: Only the final stage’s layers are included in the final image.

How It Works

You name your stages using AS name and then use COPY --from=name to pull specific files into the next stage.

# Build Stage
FROM node:20 AS build
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build

# Runtime Stage
FROM nginx:alpine
COPY --from=build /app/dist /usr/share/nginx/html

Docker Networking

Docker’s networking subsystem is pluggable, using drivers. Several drivers exist by default and provide core networking functionality.

Core Network Drivers

1. bridge (Default): The default network driver. If you don’t specify a driver, this is the type of network you are creating. Bridge networks are usually used when your applications run in standalone containers that need to communicate.
2. host: Removes network isolation between the container and the Docker host, and uses the host’s networking directly. The container does not get its own IP-address allocated. For instance, if you run a container that binds to port 80 and you use host networking, the container’s application is available on port 80 on the host’s IP address.
3. none: For this container, disable all networking. Usually used in conjunction with a custom network driver.
4. overlay: Connects multiple Docker daemons together and enables swarm services to communicate with each other. This is for multi-host networking.
5. ipvlan / macvlan: Give you direct control over IPv4 and IPv6 addressing, allowing you to assign a MAC address to a container, making it appear as a physical device on your network.

Key Networking Concepts

• DNS Resolution: Docker has a built-in DNS server that allows containers on the same user-defined network to resolve each other by name.
• Port Mapping: -p 8080:80 maps port 8080 on the host to port 80 in the container.

Docker Persistence

By default, all files created inside a container are stored on a writable container layer. This means the data is lost when the container is deleted. To persist data, Docker provides two main options: Volumes and Bind Mounts.

1. Volumes

Volumes are the preferred mechanism for persisting data. They are managed by Docker and stored in a part of the host filesystem that is managed by Docker (/var/lib/docker/volumes/ on Linux).

• Managed by Docker: You don’t need to worry about the underlying host directory structure.
• Portable: Easier to back up or migrate than bind mounts.
• Performance: Better performance on Docker Desktop (Windows/Mac) compared to bind mounts.
• Safety: New volumes can have their content pre-populated by a container.

2. Bind Mounts

Bind mounts have been available since the early days of Docker. They mount a specific file or directory on the host machine into a container.

• Host Dependent: Relies on the host machine having a specific directory structure.
• Direct Access: The host can read and modify the files directly. Useful for development (e.g., mounting source code into a container to see live changes).
• Permissions: Can be tricky as the container might require specific UID/GIDs that don’t match the host user.

3. tmpfs Mounts

Stored only in the host system’s memory. When the container stops, the tmpfs mount is removed, and files written there are discarded. Useful for sensitive information or non-persistent state that needs high performance.

Docker Security

Security in Docker is about minimizing the attack surface and preventing “container escapes” where an attacker gains access to the host.

1. Principle of Least Privilege (Non-Root User)

By default, Docker containers run as root. If a container is compromised, the attacker has root access inside the container, making it much easier to exploit kernel vulnerabilities and escape to the host. Best Practice: Create a dedicated user and group in your Dockerfile and switch to it using the USER instruction.

2. Image Scanning

Use tools like docker scout, Trivy, or Snyk to scan your images for known vulnerabilities (CVEs) in your base images and dependencies.

3. Minimize Image Size (Distroless)

Smaller images have fewer binaries (like curl, apt, or sh) that an attacker can use. Using “Distroless” images or minimal bases like Alpine Linux is a key security measure.

4. Read-Only Filesystems

If your application doesn’t need to write to the filesystem, run the container with --read-only. This prevents attackers from downloading and executing malicious scripts.

5. Secret Management

Never bake secrets (API keys, passwords) into the Docker image layers. Use environment variables (carefully), Docker Secrets, or vault providers.

Dockerfile Instructions

Writing efficient Dockerfiles requires understanding the specific behavior of each instruction.

1. CMD vs. ENTRYPOINT

Both instructions define what command runs when a container starts, but they behave differently:

• ENTRYPOINT: Sets the main executable for the image. It is not easily overridden by command-line arguments. Use this for containers that should act like executables (e.g., a CLI tool).
• CMD: Sets default arguments for the ENTRYPOINT or a default command. It is easily overridden by providing arguments to docker run.

The Pattern: Usually, you use ENTRYPOINT for the binary and CMD for the default flags.

2. COPY vs. ADD

• COPY: The preferred instruction. It simply copies files or directories from the host into the container.
• ADD: Does everything COPY does, but also has extra features: it can fetch files from remote URLs and automatically extract tarballs. Best Practice: Use COPY unless you explicitly need ADD’s extra features to keep the build process transparent.

3. Shell vs. Exec Form

Most instructions can be written in two ways:

• Shell form: CMD executable param1 (runs as /bin/sh -c)
• Exec form: CMD ["executable", "param1"] (preferred; runs the executable directly, allowing it to receive Unix signals like SIGTERM).