OpenHPC on LQ1 cluster

The HPC software on the Fermilab LQ1 cluster is packaged and distributed by the OpenHPC project. We are currently providing OpenHPC release v1.3 for Red Hat Enterprise 7.7

Installed software components

Available software components are easily configured using the Lua lmod system which modifies the PATH and LD_LIBRARY_PATH (bash) shell environment variables and sets any other needed variables. Lmod also guards against a user having conflicting software packages (e.g. gnu8 compiler and an MPI built with clang) in your shell environment. More information on using lmod is available in the Introduction to lmod.

Quick introduction to using Lmod

You can list all of the software components available using the spider option:

$ module spider # (spyder is also acceptable spelling)

[output suppressed]

There is also an avail option, however, this option, unlike spider will only show available modules that do not conflict with your currently configured options. Assuming we do not have anything already configured with lmod, the avail option will display mainly the available compilers and other utilities that do not depend upon a particular compiler. Here is an (edited) example of the listing you will see:

$ module avail

---------------------------- /opt/ohpc/pub/modulefiles ------------------------------

EasyBuild/3.9.2 cmake/3.14.3 gnu8/8.3.0 llvm5/5.0.1 autotools

autotools gnu/5.4.0 hwloc/2.0.3 papi/5.7.0 singularity/3.2.1

charliecloud/0.9.7 gnu7/7.3.0 intel/19.0.4.243 pmix/2.2.2 valgrind/3.15.0

The load command will enable a software package within your shell environment. If there is only a single package version available, it suffices to use the package name, e.g. gnu8, without specifying the particular version: gnu8/8.3.0. The following loads autotools, cmake, and gnu8:

$ module load autotools cmake gnu8

Currently loaded modules can be displayed with list:

$ module list

Currently Loaded Modules:

1) autotools 2) cmake/3.14.3 3) gnu8/8.3.0

If a package is no longer needed, it can be unloaded:

$ module unload autotools

$ module list

Currently Loaded Modules:

1) cmake/3.14.3 2) gnu8/8.3.0

Running avail again will now show available modules that have gnu8 as a dependency:

$ module avail

 

--------------- /opt/ohpc/pub/moduledeps/gnu8 -----------------

R/3.5.3 impi/2019.4.243 mpich/3.3 openblas/0.3.5 plasma/2.8.0 scotch/6.0.6

gsl/2.5 likwid/4.3.4 mvapich2/2.3.1 openmpi3/3.1.4 py2-numpy/1.15.3 superlu/5.2.1

hdf5/1.10.5 metis/5.1.0 ocr/1.0.1 pdtoolkit/3.25 py3-numpy/1.15.3

We can now load an MPI package:

$ module load openmpi3

$ module list

Currently Loaded Modules:

1) cmake/3.14.3 2) gnu8/8.3.0 3) openmpi3/3.1.4

We can use swap to change to another MPI implementation:

$ module swap openmpi3 mvapich2

$ module list

Currently Loaded Modules:

1) cmake/3.14.3 2) gnu8/8.3.0 3) mvapich2/2.3.1

The purge command will unload all current modules:

$ module purge

$ module list

No modules loaded

This is useful at the beginning of batch scripts to prevent the batch shell from unintentionally inheriting a module environment from the submission shell.

Launching MPI processes

Near the beginning of your batch script, prior to launching an MPI process you should ensure only software modules required by the batch script have been loaded, for example if using gnu8 and openmpi:

$ module purge

$ module load gnu8 openmpi3

There are two common mechanisms for starting up MPI either use mpirun or srun. An mpirun command is provided by each MPI implementation, and is specific to that implementation. The commands provided by openmpi, mvapich, and Intel impi are all slurm “aware”. They will attempt to use slurm interfaces to distribute and start MPI binaries. In addition, slurm has the srun command that is able to start up MPI. The following command will list APIs srun supports for MPI.

$ srun --mpi=list

srun: MPI types are...

srun: pmi2

srun: pmix_v3

srun: pmix

srun: none

srun: openmpi

srun: pmix_v2

The table below lists recommended launchers for the different MPI implementations. These combinations have been proved to work. Combinations that are not listed either fail, or do not properly launch MPI.

We recommend using srun to launch all three MPI implementations since it is then possible to use command line options to specify the distribution and binding of MPI ranks to CPU cores and local memory. The options are described in the srun documentation. Careful specification of the process affinities is especially important when running MPI in the hybrid approach combining MPI with thread parallelism. TU Dresden has a nice compendium illustrating different srun distribution and binding options for MPI.

Binding and distribution of tasks

There's a good description of MPI process affinity binding and srun here: Click here

 

Reasonable affinity choices by partition types on the Fermilab LQCD clusters are:

 

Intel (lq1) --distribution=cyclic:cyclic --cpu_bind=sockets --mem_bind=no

Fermi National Accelerator Laboratory

Managed by Fermi Research Alliance, LLC

for the U.S. Department of Energy Office of Science

Security, Privacy, Legal