Using Apptainer and Docker Containers

What is Apptainer

Apptainer is a containerization solution designed for high-performance computing cluster environments.  It allows a user on an HPC resource to run an application using a different operating system than the one provided by the cluster.  For example, the application may require Ubuntu but the cluster OS is CentOS.  Conceptually, it is similar to other container software such as Docker, but is designed with several important differences that make it more suited for HPC environments.  

• Encapsulation of the environment
• Containers are image based
• No user contextual changes or root escalation allowed
• No root owned daemon processes

Finding Images

Apptainer can run images from a variety of sources, including both a flat image file or a Docker image from Docker Hub.

Docker Hub

Publically available Docker images can be found at Docker Hub. For convenience, HCC also provides a set of images on Docker Hub known to work on HCC resources.  

Available Images at HCC

The following table lists the currently available images and the command to run the software.

5 10 20 30 40 All Rows 1 2 3 4 5

5 10 20 30 40 All Rows 1 2 3 4 5

Use images on HCC resources

To use Apptainer on HCC machines, first load the apptainer module. Apptainer provides a few different ways to access the container. Most common is to use the exec command to run a specific command within the container; alternatively, the shell command is used to launch a bash shell and work interactively.  Both commands take the source of the image to run as the first argument.  The exec command takes an additional argument for the command within the container to run.

Finally, pass any arguments for the program itself in the same manner as you would if running it directly.  For example, the Spades Assembler software is run using the Docker image unlhcc/spades and via the command spades.py. To run the software using Apptainer, the commands are:

module load apptainer
apptainer exec docker://unlhcc/spades spades.py <spades arguments>

Use images within a SLURM job

Using Apptainer in a SLURM job is similar to how you would use any other software within a job. Load the module, then execute your image:

#!/bin/bash
#SBATCH --time=03:15:00          # Run time in hh:mm:ss
#SBATCH --mem-per-cpu=4096       # Maximum memory required per CPU (in megabytes)
#SBATCH --job-name=apptainer-test
#SBATCH --error=/work/[groupname]/[username]/job.%J.err
#SBATCH --output=/work/[groupname]/[username]/job.%J.out

module load apptainer
apptainer exec docker://unlhcc/spades spades.py <spades arguments>

Create a custom image

Custom images can be created locally on your personal machine and added to Docker Hub for use on HCC clusters. More information on creating custom Docker images can be found in the Docker documentation.

You can create custom Docker image and use it with Apptainer on our clusters. Apptainer can run images directly from Docker Hub, so you don’t need to upload anything to HCC. For this purpose, you just need to have a Docker Hub account and upload your image there. Then, if you want to run the command “mycommand“ from the image “myimage”, type:

module load apptainer
apptainer exec docker://myaccount/myimage mycommand

where “myaccount” is your Docker Hub account.

In case you see the error ERROR MANIFEST_INVALID: manifest invalid when running the command above, try:

module load apptainer
unset REGISTRY
apptainer exec docker://myaccount/myimage mycommand

All the Dockerfiles of the images we host on HCC are publicly available here. You can use them as an example when creating your own image.

Add packages to an existing image

Alternatively, instead of building an image from scratch, you can start with an HCC-provided image as the base for your Dockerfile (i.e. FROM unlhcc/spades) and add any additional packages you desire.

Unfortunately it’s not possible to create one image that has every available Python package installed for logistical reasons.  Images are created with a small set of the most commonly-used scientific packages, but you may need others.  If so, you can install them in a location in your $WORK directory and set the PYTHONPATH variable to that location in your submit script.  The extra packages will then be “seen” by the Python interpreter within the image.  To ensure the packages will work, the install must be done from within the container via the apptainer shell command.  For example, suppose you are using the tensorflow-gpu image and need the packages nibabel and tables.  First, run an interactive SLURM job to get a shell on a worker node.

srun --pty --mem=4gb --qos=short --gres=gpu --partition=gpu $SHELL

After the job starts, the prompt will change to indicate you’re on a worker node.  Next, start an interactive session in the container.

module load apptainer
apptainer shell docker://unlhcc/tensorflow-gpu

This may take a few minutes to start.  Again, the prompt will change and begin with Apptainer to indicate you’re within the container.

Next, install the needed packages via pip to a location somewhere in your work directory.  For example, $WORK/tf-gpu-pkgs.  (If you are using Python 3, use pip3 instead of pip).

export LC_ALL=C
pip install --system --target=$WORK/tf-gpu-pkgs --install-option="--install-scripts=$WORK/tf-gpu-pkgs/bin" nibabel tables

You should see some progress indicators, and a “Successfully installed..." message at the end.  Exit both the container and the interactive SLURM job by typing exit twice.  The above steps only need to be done once per each image you need additional packages for.   Be sure to use a separate location for each image’s extra packages.

To make the packages visible within the container, you’ll need to add a line to the submit script used for your Apptainer job.  Before the lines to load the apptainer module and run the script, add a line setting the PYTHONPATH variable to the $WORK/tf-gpu-pkgs directory. For example,

#!/bin/bash
#SBATCH --time=03:15:00          # Run time in hh:mm:ss
#SBATCH --mem-per-cpu=4096       # Maximum memory required per CPU (in megabytes)
#SBATCH --job-name=apptainer-test
#SBATCH --partition=gpu
#SBATCH --gres=gpu
#SBATCH --error=/work/[groupname]/[username]/job.%J.err
#SBATCH --output=/work/[groupname]/[username]/job.%J.out
 
export PYTHONPATH=$WORK/tf-gpu-pkgs
module load apptainer
apptainer exec docker://unlhcc/tensorflow-gpu python /path/to/my_tf_code.py

The additional packages should then be available for use by your Python code running within the container.

What if I need a specific software version of the Apptainer image?

You can see all the available versions of the software built with Apptainer in the table above. If you don’t specify a specific sofware version, Apptainer will use the latest one. If you want to use a specific version instead, you can append the version number from the table to the image. For example, if you want to use the Apptainer image for Spades version 3.11.0, run:

apptainer exec docker://unlhcc/spades:3.11.0 spades.py