-
Notifications
You must be signed in to change notification settings - Fork 8
How to install using container
Container is a software packaging method that allows the software to run on various OS. Yandasoft comes both as source code in this GitHub project, and as pre-built packages that can be downloaded from our account in DockerHub. The packaging technology that we use is Docker. The package is called an image. A container is an instance built from the image. The container has Yandasoft inside it, complete with all dependencies that it requires to run. Installing the software this way is far simpler than building it from source code, so this is the method that we recommend.
If you are installing Yandasoft on your own computer, you can use either Docker or Singularity. They both works for the same Docker image.
If you are installing on shared computer (HPC), please use Singularity. Using container on HPC is a rather complicated matter, because of these reasons:
- User privilege. This is directly related to security. Running Docker daemon requires root privilege, whereas Singularity can be used as an ordinary user.
- Job scheduling
- Parallel computation, in particular MPI implementation. More discussion can be found in this section.
Singularity was designed to tackle these issues. Jump to Singularity section using this link.
The steps for installation using Docker have been tested on the following host systems running on personal machines (ie. non HPC):
- Linux (Ubuntu18.04)
- MacOS 10.15 "Catalina"
- Windows 10
It should work from other platforms that can run Docker.
This step is necessary only if you have not installed Docker in your computer.
The centrepiece of this virtualisation method. An overview can be found here: https://docs.docker.com/install/overview/. Procedure for a particular platform can be found in the sub-directories. Installation instruction for specific OS can be found here:
- Ubuntu: https://docs.docker.com/install/linux/docker-ce/ubuntu/
- Mac: https://docs.docker.com/docker-for-mac/install/
- Windows: https://docs.docker.com/v17.09/docker-for-windows/install/
To avoid using sudo all the time, we need to add Docker group if it does not already exist, and then add the current user to the group.
sudo groupadd docker
sudo gpasswd -a $USER docker
newgrp docker
Note that Docker will first automatically download the image, which in turn will be run as container inside your system. You will run Yandasoft from inside that container.
For convenience, make local directory for Yandasoft. Later, this directory will be linked to the container, so that executables inside the container can "see" files in this directory.
mkdir yandasoft
A pre-built image is already stored in DockerHub. Simply pull one of the images to our computer using this command:
docker pull csirocass/yandasoft:1.1-mpich
The image will be now pulled to our computer. This takes a while, but it's still faster than building everything from scratch. When it's finished, we can see that the image now resides locally by typing this:
docker images
The output will be something like this:
REPOSITORY TAG IMAGE ID CREATED SIZE
...
csirocass/yandasoft 1.1-mpich 3ea2ce3f5de4 4 days ago 1.33GB
...
We want interactive Docker session for the container (hence "-it" directive). To be able to access files in the host file system from within the container, we do bind-mounting using the "-v" directive, which maps host directory to container directory. So to map "/my_path/yandasoft" (replace this with your path) in the host to "/home" in the container, do the following.
docker run -it -v /my_path/yandasoft:/home csirocass/yandasoft:1.1-mpich /bin/bash
This command will create Docker container based on the image mentioned above, and we will be taken into the Docker environment (in this case it's Linux). A bash session from the container will replace our current local session. The meaning of the command options "-it v" is explained below.
--interactive , -i Keep STDIN open even if not attached
--tty , -t Allocate a pseudo-TTY
--volume , -v Bind mount a volume
Note that the options "-i" and "-t" are usually used together, because an interactive session works better with TTY, so they are abbreviated as "-it".
Simply go to home directory, and run ASKAP commands. Shown in the example below is running "cimager" using configuration file "config.in". Note that the file needs to be supplied separately from within the host file system.
cd /home
cimager -c config.in
To explore available commands, take a look in this directory:
ls /usr/local/bin
To end the session, simply type:
exit
This will take us out of Docker environment and back to the host OS. Note that after we exit the session, the container is no longer running, but it still exists. To list all the containers (including those that are no longer running), execute this:
docker ps -a
This will give output that looks like this:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
...
5d13369bb11f csirocass/yandasoft:1.1-mpich "/bin/bash" 27 hours ago Exited (127) 33 seconds ago romantic_spence
...
After running (and exiting) Docker for a number of times, we might end up with too many containers. For ease of management, we can remove the containers using the following command (replace "container_id" with appropriate number):
docker rm container_id
Similarly, after a while we might end up with too many images. This is how to remove an image:
docker rmi image_id
Note that if the image has dependent child image, it will not be deleted. All of its child images must be deleted first.
When we exit a container, it's still available for future use, until we explicitly delete it. We can restart and stop a container by typing this commands:
docker restart container_id
docker stop container_id
However, a container restarted this way will run in the background. For our purpose, we would like to have an interactive session, and the command for that is as follows:
docker exec -it container_id /bin/bash
where the options have the same meaning as in the "docker run" command that we used before.
--interactive , -i Keep STDIN open even if not attached
--tty , -t Allocate a pseudo-TTY
Note that the directory that was mounted at the beginning (by "docker run" command) is still mounted, so there is no need to repeat the mounting process.
Singularity uses the same Docker image hosted in DockerHub, but it converts it into Singularity container. The steps in this section have been tested on these HPC:
- Pearcey: generic MPICH, OpenMPI-4.0.2, OpenMPI-3.1.4, and OpenMPI-2.1.6
- Pawsey's HPC: Galaxy and Zeus (Cray MPICH)
Note that, Singularity and its ecosystem must be installed and set up by the administrators of HPC. User then simply loads the Singularity module.
module load singularity
User also needs to set ownership and permission for Singularity cache. For example, in Pawsey's HPC (Galaxy or Zeus), you need to do this once at the very beginning of Singularity set up (note that there is no need to do this again for subsequent use of Singularity).
mkdir -p $MYGROUP/.singularity
chown -hR $USER:$PAWSEY_PROJECT $MYGROUP/.singularity
find $MYGROUP/.singularity -type d -exec chmod g+s {} \;
Further reading for Pawsey's HPC can be found here
This may take a long time, so it's better to pull the image from compute node rather than from login node, for example by doing this first:
salloc -t 60
Note that a number of Docker images with specific MPI implementation are available. Please pull the the one that matches your machine.
| MPI | Pull command | Note |
|---|---|---|
| OpenMPI 4 | singularity pull docker://csirocass/yandasoft:1.1-openmpi4 |
The latest OpenMPI |
| OpenMPI 3 | singularity pull docker://csirocass/yandasoft:1.1-openmpi3 |
No longer supported by OpenMPI |
| OpenMPI 2 | singularity pull docker://csirocass/yandasoft:1.1-openmpi2 |
No longer supported by OpenMPI |
| MPICH | singularity pull docker://csirocass/yandasoft:1.1-mpich |
|
| Cray MPICH | singularity pull docker://csirocass/yandasoft:1.1-galaxy |
For Galaxy and Zeus in Pawsey |
A local Singularity image will then be created. Singularity version 3 and beyond will create file with sif extension. For example, if the image is for MPICH, then the file will look like this:
yandasoft_mpich.sif
whereas previous versions will create file with simg extension:
yandasoft_mpich.simg
From users' point of view, there is no difference between the two formats.
This is because the pull process is incomplete. Unfortunately, this happens silently, so the error appears at later stage. When this happens, repeat the pull after clearing the cache using this command:
singularity cache clean
Remember to check that you actually have enough disk space for the image!
Running the image as interactive shell is even simpler than equivalent Docker's command. Again, it's better to do this on compute node rather than login node (using "salloc").
singularity shell yandasoft_mpich.sif
Note that local directory is automatically mounted in Singularity, whereas in Docker, this must be explicitly stated in "-v" option.
Complexity when running executable within Singularity container arises due to job scheduler (eg. SLURM) and MPI implementation for parallel computation (eg. MPICH). Each HPC system has its own job scheduler and MPI implementation. For example, to run this basic Yandasoft tutorial on HPC called Galaxy, we would need the following batch file (called "dirty.sbatch" in the tutorial).
#!/bin/bash -l
## Batch file for Galaxy
#SBATCH --ntasks=305
#SBATCH --time=01:00:00
#SBATCH --partition=workq
#SBATCH --export=NONE
module load singularity
module load cray-mpich
export OMP_NUM_THREADS=1
export SINGULARITY_CACHE_DIR=/group/askap/my_account/singularity/cache
srun --export=all -n 305 singularity exec yandasoft_galaxy.sif cimager -c dirty.in
On another machine, called Pearcey, when MPICH module is loaded, the batch file looks like this:
#!/bin/bash -l
#SBATCH --ntasks=5
#SBATCH --time=01:00:00
#SBATCH --export=NONE
module load singularity
module load mpich/3.3.0
mpirun -n 5 singularity exec yandasoft_mpich.sif cimager -c dirty.in
When OpenMPI-4.0.2 is used on the same machine instead of MPICH, the batch file looks like this:
#!/bin/bash -l
#SBATCH --ntasks=5
#SBATCH --time=01:00:00
#SBATCH --export=NONE
module load singularity
module load openmpi/4.0.2
mpirun -n 5 singularity exec yandasoft_mpich.sif cimager -c dirty.in
Note that the contents of the batch files are provided as a guide only. Please check whether the details are correct. Version numbers will certainly change after a while.
The two major MPI implementations that concern us are MPICH and OpenMPI. One of the most important difference between the two as far as our deployment is concerned is that MPICH has uniform Application Binary Interface (ABI) that ensures compatibility across vendors and versions. This makes its adoption particularly attractive. On the other hand, compatibility in OpenMP is more complicated (see this page). A more through comparison can be found in this StackOverflow page.
This leads to a rather lengthy discussion, so please see this SLURM webpage if you are interested. As a short summary, in order to connect the two, most modern MPI implementations use an API called PMIx (Process Management Interface – Exascale. Note its use in "srun" in Galaxy by means of the option: "–mpi=pmi2" to explicitly engage the API. Available options can be seen by typing this command:
srun --mpi=list
which gives:
srun: MPI types are...
srun: none
srun: openmpi
srun: pmi2
Note that "none" is the default.
In Pearcey, "mpirun" already takes care of this.