Defining Services in docker-compose.yml

At the heart of a Docker Compose configuration is the services top-level key in your docker-compose.yml file. This section acts as a container for defining each independent component, or "service," that constitutes your application. Think of a service as the blueprint for a container within your multi-container setup. Each service will typically run in its own container, isolated yet capable of communicating with other services defined in the same file, which we will cover later.

Let's consider a common scenario in Machine Learning development: you might have a web application (perhaps a Flask or FastAPI API serving predictions) and a separate database service (like PostgreSQL or Redis) to store model metadata or user information. In Docker Compose, each of these would be defined as a distinct service under the services key.

The name you give each service (e.g., api, db, worker) is significant. It becomes a hostname within the Docker network created by Compose, allowing services to discover and communicate with each other easily.

To define a service, you start by specifying its name, followed by its configuration details. The most fundamental piece of configuration is telling Compose what container image to use for the service. There are two primary ways to do this:

1. Using a Pre-built Image (image): If your service relies on a standard, publicly available image (like the official Python image, a database image like postgres, or a pre-built ML framework image from Docker Hub), you use the image directive. You specify the image name and tag, just as you would with docker pull or docker run.

   # docker-compose.yml
   version: '3.8' # Specify Compose file version (optional but recommended)
   
   services:
     database:
       image: postgres:14-alpine # Use the official PostgreSQL 14 Alpine image
       # ... other configuration for the database service ...
   
     redis_cache:
       image: redis:7-alpine # Use the official Redis 7 Alpine image
       # ... other configuration for the Redis service ...
   

2. Building from a Dockerfile (build): If your service requires a custom environment, such as your specific ML application code, dependencies defined in requirements.txt, or particular configurations, you'll typically build an image from a Dockerfile. The build directive tells Compose where to find the build context (the directory containing the Dockerfile and any files needed for the build).

   • If the Dockerfile is named Dockerfile and located in the same directory as docker-compose.yml, you can simply specify the context path:

     # docker-compose.yml
     version: '3.8'
     
     services:
       ml_api:
         build: . # Build the image using Dockerfile in the current directory
         # ... other configuration for the API service ...
     

   • If your project structure is more complex, or your Dockerfile has a different name, you can provide more details within the build directive:

     # docker-compose.yml
     version: '3.8'
     
     services:
       training_job:
         build:
           context: ./training_module # Path to the directory containing Dockerfile and source code
           dockerfile: Dockerfile.train # Specify a custom Dockerfile name
         # ... other configuration for the training service ...
     

It's common to mix these approaches in a single docker-compose.yml file. You might build your custom API service using build while pulling a standard database image using image.

Here is a combined example:

# docker-compose.yml
version: '3.8'

services:
  # Service for the ML prediction API
  prediction_api:
    build: ./api # Assumes ./api/Dockerfile exists
    # Configuration like ports, volumes, env vars will be added here later

  # Service for the database
  metadata_db:
    image: postgres:14-alpine
    # Configuration like volumes, env vars for DB setup will be added here later

This simple structure defines two services: prediction_api, which will be built from the Dockerfile located in the ./api directory, and metadata_db, which will use the pre-built postgres:14-alpine image.

The docker-compose.yml file defines services, specifying whether to build them from a local Dockerfile (prediction_api) or pull a pre-existing image (metadata_db).

While specifying the image source (image or build) is fundamental, defining a service involves much more. In the following sections, we will discuss how to configure networking between these services, manage persistent data using volumes, expose ports, set environment variables, and control the startup order and dependencies between services, all within the docker-compose.yml file under each service definition.