Docker Compose

Up to this point, we have been looking at single-container applications - small units of code that are containerized, executed ad hoc to generate or read a file, then exit on completion. But what if we want to do something more complex? For example, what if our goal is to orchestrate a multi-container application consisting of, e.g., a Flask app, a database, a front-end service, and more.

Docker compose is a tool for managing multi-container applications. A YAML file is used to define all of the application service, and a few simple commands can be used to spin up or tear down all of the services.

In this module, we will get a first look at Docker compose. After going through this module, students should be able to:

• Translate Docker run commands into YAML files for Docker compose

• Run commands inside ad hoc containers using Docker compose

• Manage small software systems composed of more than one script, and more than one container

• Copy data into and out of containers as needed

Another Script, Another Container

We have been working a lot with a script for reading in and analyzing a simple text file. Let’s quickly write a new script to generate that data, then we will package it into its own container. Consider the following script for generating sample number like we have been working with:

#!/usr/bin/env python3

import random
import sys
import names

'''
Collatz Generator Code for CS 401

Author: Andrew Solis
'''

NUM = 20

def main():


   for i in range(NUM):

      rand_num = random.randint(1, 1000)

      with open(sys.argv[1], 'a') as file:
            file.write( str(rand_num ) + '\n')


   print(f'Data written to {sys.argv[1]}')

   print(f'Codename { names.get_first_name() }')

if __name__ == '__main__':
   main()

Attention

This script requires a python package called names. This can be installed from your terminal by issuing the command pip install names. If you get an error then you can create a virtual environment first by running the following commands:

python3 -m venv venv

# linux/mac

source venv/bin/activate

# Windows

venv\Scripts\activate

Then you can install the package with the command pip install names. If you do not know what virtual environments are do not worry, we will cover them later in this course. If you are still having issues then reach other to the professor for help.

Copy that into a file called gen_collatz.py, save it, make it executable, and test it. You’ll find that this script requires a command line argument. Meaning we have to invoke it AND pass some information on the command line in order to get it to work. In this case, it is expecting the name of the output file.

# copy contents into file called ``gen_collatz.py`` and save
[terminal]$ chmod +rx gen_collatz.py
[terminal]$ python gen_collatz.py
Traceback (most recent call last):
  File "./gen_collatz.py", line 29, in <module>
    main()
  File "./gen_collatz.py", line 25, in main
    with open(sys.argv[1], 'w') as o:
IndexError: list index out of range

# need to provide output filename on command line
[terminal]$ python gen_collatz.py data.txt
Data written to data.txt.
Codename Sharon
[terminal]$ ls
data.txt  Dockerfile  gen_collatz.py  input.txt  collatz.py
[terminal]$ head -n11 data.txt
409
1
371
622
6
229
617
236
778
165
306

Containerizing this script should be easy enough - we already worked through containerizing another very similar script. But, lets say we need a different dependency: the Python3 names library.

To make things a little more clear, rename the existing Dockerfile as Dockerfile-program, and make a copy of it called Dockerfile-gen.

[terminal]$ mv Dockerfile Dockerfile-program
[terminal]$ cp Dockerfile-program Dockerfile-gen
[terminal]$ ls
Dockerfile-gen     Dockerfile-program collatz.py
data.txt           gen_collatz.py     input.txt

Edit Dockerfile-gen as follows:

FROM ubuntu:24.04

RUN apt update && \
    apt upgrade -y && \
    apt install -y python3.12 python3-pip vim python3.12-venv

RUN python3 -m venv venv

ENV PATH="/venv/bin:$PATH"

RUN pip install names

COPY gen_collatz.py /code/gen_collatz.py

RUN chmod +rx /code/gen_collatz.py

ENV PATH="/code:$PATH"

Now that we have a Dockerfile named something other than the default name, we need to modify our command line a little bit to build it:

[terminal]$ docker build -t username/gen_collatz:1.0 -f Dockerfile-gen .

After the image is successfully built, change directories to a new folder just to be sure you are not running the local scripts or looking at the local data. Then, test the container as follows:

[terminal]$ mkdir test
[terminal]$ cd test
[terminal]$ ls
[terminal]$ docker run --rm username/gen_collatz:1.0 gen_collatz.py data.txt
Data written to data.txt
Codename Alex

If you list your local files, can you find data.txt? No! This is because whatever data generated inside the container is lost when the container completes its task. What we need to do is use the -v flag to mount a directory somewhere inside the container, write data into that directory, then the data will be captured after the container exists. For example:

[terminal]$ docker run --rm -v $PWD:/data username/gen_collatz:1.0 gen_collatz.py /data/data.txt
Data written to data.txt.
Codename Sandra

Note

To reiterate, because we mounted our current location as a folder called “/data” (-v $PWD:/data), and we made sure to write the output file to that location in the container (gen_collatz.py /data/data.txt), then we get to keep the file after the container exits, and it shows up in our current location ($PWD).

EXERCISE

Spend a few minutes testing both containers. Be sure you can generate data with one container, then read in and analyze the same data with the other. Data needs to persist outside the containers in order to do this.

Write a Compose File

Docker compose works by interpreting rules declared in a YAML file (typically called docker-compose.yml). The rules we will write will replace the docker run commands we have been using, and which have been growing quite complex. For example, the commands we used to run our JSON parsing scripts in a container looked like the following:

[terminal]$ docker run --rm -v $PWD:/data username/gen_collatz:1.0 gen_collatz.py /data/data.txt
[terminal]$ docker run --rm -v $PWD/data.txt:/data/data.txt username/collatz:1.0 collatz.py /data/data.txt

The above docker run commands can be loosely translated into a YAML file. Navigate to the folder that contains your Python scripts and Dockerfiles, then create a new empty file called docker-compose.yml:

[terminal]$ pwd
/home/asolis/docker-exercise
[terminal]$ touch docker-compose.yml
[terminal]$ ls
Dockerfile-gen     collatz.py         docker-compose.yml
Dockerfile-program gen_collatz.py     test/

Next, open up docker-compose.yml with your favorite text editor and type / paste in the following text:

---

services:
   gen-data:
      build:
            context: ./
            dockerfile: ./Dockerfile-gen
      image: username/gen_collatz:1.0
      volumes:
            - ./test:/data
      command: gen_collatz.py /data/data.txt
   analyze-data:
      build:
            context: ./
            dockerfile: ./Dockerfile-program
      depends_on:
            - gen-data
      image: username/collatz:1.0
      volumes:
            - ./test:/data
      command: collatz.py /data/data.txt

Warning

The highlighted lines above may need to be edited with your username in order for this to work. See instructions below.

The services section defines the configuration of individual container instances that we want to orchestrate. In our case, we define two called gen-data for the gen_collatz functionality, and analyze-data for the collatz program functionality.

Each of those services is configured with its own Docker image, a mounted volume (equivalent to the -v option for docker run), and a default command to run.

Note

The top-level services keyword shown above is just one important part of Docker compose. To learn about aboth important parts of a compose file please visit the Docker Compose Docs.

Running Docker Compose

The Docker compose command line tool follows the same syntax as other Docker commands:

docker compose <verb> <parameters>

Just like Docker, you can pass the --help flag to docker compose or to any of the verbs to get additional usage information. To get started on the command line tools, try issuing the following two commands:

[terminal]$ docker compose version
[terminal]$ docker compose config

The first command prints the version of Docker compose installed, and the second searches your current directory for docker compose.yml and checks that it contains only valid syntax.

To run one of these services, use the docker compose run verb, and pass the name of the service as defined in your YAML file:

[terminal]$ ls test/     # currently empty
[terminal]$ docker compose run gen-data
Data written to /data/data.txt.
Codename Alice.
[terminal]$ ls test/
data.txt               # new file!
[terminal]$ docker compose run analyze-data
Collatz
=======
...

Now we have an easy way to run our ad hoc services consistently and reproducibly. Not only does docker compose.yml make it easier to run our services, it also represents a record of how we intend to interact with this container.

Essential Docker Compose Command Summary

Command	Usage
docker compose version	Print version information
docker compose config	Validate docker compose.yml syntax
docker compose up	Spin up all services
docker compose down	Tear down all services
docker compose build	Build the images listed in the YAML file
docker compose run	Run a container as defined in the YAML file

Additional Resources

• Docker Compose Docs