Using Singularity to containerise a Scientific Model

I was asked to install the parallel ice sheet model (PISM) by one of our researches. PISM is modern high-order physics ice sheet model that is written in C++ and uses MPI and PETSc. All our Linux machines are LCFG managed Scientific Linux 7 machines. PISM is built from source using cmake. Unfortunately, cmake cannot find Open MPI the way it is packaged on SL7. This is a good opportunity to try using singularity to containerise a scientific model.

Singularity is container platform designed for scientific software. It can be used to package complex models with their dependencies in a single file that can be run on a laptop or a HPC cluster. It is very useful to define an environment that can be used again in future to ensure reproducible model runs. One of the key features from my perspective is that it is also secure. Singularity containers cannot gain additional privileges since the user inside the container is the same user as outside. No special privileges are required to run a singularity container. However, users need to have elevated privileges (which they gain via sudo) to build a container.

Compiling Open MPI, PETSc and PISM takes a while. I decided to split the problem up into three containers, each depending on the previous one:

the base container provides MPI
PETSc is built in a second container based on the base container
PISM is built in a container based on the PETSc container

This structure allows the containers to be reused for other projects. It also makes it easier to test them.

The container definition files contain instructions on how to build them. I use Ubuntu 20.04 as the base system. Additional software gets installed into the container using apt. Source code is downloaded using wget and git.

PISM uses MPI. Singularity does support MPI but the version of MPI inside the container has to be same as outside the container. In addition to OpenMPI I had to also compile a matching version of the portable hardware locality (hwloc) library.

The containers are built using a `Makefile <https://git.ecdf.ed.ac.uk/uoe-gits/singularity-pism/blob/master/Makefile>`__. The model binaries are installed into the /opt/pism directory.

Once the containers are built a simple test run confirms that the model is working:
module load mpi/openmpi-x86_64 mpirun -n 4 singularity exec --bind /scratch pism.sif /opt/pism/bin/pismv -test G -y 200
The first line loads the Open MPI environment. By default singularity exposes the user's home directory. Other directories can be exposed using the --bind option. The example above exposes the /scratch directory. The containerised model will run in the present working directory if that is accessible to the container (exposed via the --bind option).

PISM comes with some examples including a setup for the Greenland Ice Sheet. Input data files are generated using the preprocess script running on the host system.
cd examples/std-greenland ./preprocess.sh
The containerised model is run using the example script by setting an environment variable:
export PISM_EXEC="/usr/bin/singularity exec --bind /scratch /PATH/TO/pism.sif  /opt/pism/bin/pismr" ./spinup.sh 4 const 10000 20 sia g20km_10ka.nc

The container definition files can be found in my git repository.